antp.be forum

Hello,
I have been getting 403 forbidden error with the IAFD script, changing the User Agent in [GetScriptWin] is not helping, anyone else with this issue?
Help!!

I understand this is because they have installed an anti-scraping software or something limiting this but any other workaround?
Anyone has an example of GetPage5 changing the user agent? is it possible?
Thank you!

maybe you are right in pointing that error but il should be helpfully to say what movie you are searching.
https://www.netsons.com/manage/knowledg ... KwZISrLAmw

Doesn't matter much which, it doesn't work with any movie, it happens on "GetPage" function.
I have tried with movies that used to work, still the same.

robbdj75 wrote: 2025-02-11 18:38:40 Doesn't matter much which, it doesn't work with any movie, it happens on "GetPage" function.
I have tried with movies that used to work, still the same.

I'm sorry, I tried, but I don't know how to help you.

At some point if sites block the retrieval of their data, I can't do anything against it, it is their right to refuse it

antp wrote: 2025-02-17 18:28:53 At some point if sites block the retrieval of their data, I can't do anything against it, it is their right to refuse it

antp wrote: 2025-02-17 18:28:53 At some point if sites block the retrieval of their data, I can't do anything against it, it is their right to refuse it

It could be a problem about user gent, isnt'it?

User agent is "Mozilla/5.0 (compatible; Ant Movie Catalog)", and browsing the site with that user agent set in the browser seems to work.
So it is probably a more generic anti-bot system that they use.
And it also means that just changing the user agent will not be enough, as said in the first post...

antp wrote: 2025-02-20 18:09:06 ... just changing the user agent will not be enough, as said in the first post...

I saw it... sorry, I forgot to say that in robots.txt there is a stop to every spider.
May be that robots.txt could be bypassed?
This is: https://www.iafd.com/robots.txt

• # No sucking our data. You're welcome to use our homepage as a reference
  # and our glossary and such... but no data...
  # read our TOS for more details http://www.iafd.com/
  
  
  User-agent: *
  Disallow: /*-responsive.asp
  Disallow: /*-responsive.rme
  Disallow: /classified/
  Disallow: /scripts/
  Disallow: /java/
  Disallow: /snitzf/
  Disallow: /snitzfor/
  Disallow: /stats/
  Disallow: /shopclick.asp
  Disallow: /person.asp
  Disallow: /title.asp
  Disallow: /banman/
  Disallow: /*Zuleidy*
  Disallow: /*zuleidy*
  Disallow: /*jhennifers*
  Disallow: /*-Lupe*
  Disallow: /*/Lupe*
  Disallow: /*-lupe*
  Disallow: /*/lupe*
  Disallow: /*jkenney*
  Disallow: /*angelika_cz*
  Disallow: /buymovie.rme/
  Disallow: /reviewjump.rme/
  Disallow: /galleryclick
  Disallow: /random
  Disallow: /*joaosei*
  
  
  User-agent: CCBot
  Disallow: /
  
  User-agent: Fasterfox
  Disallow: /
  
  user-agent: AhrefsBot
  disallow: /
  
  # No sucking our stuff
  User-agent: Teleport Pro*
  Disallow: /
  
  # No sucking our stuff
  User-agent: Wget*
  Disallow: /
  
  sitemap: https://www.iafd.com/sitemap-master.xml

The robots.txt is just an indication for robots so they know what they can browse or not, but it is up to each of them to follow it or not.