IMDB Script seems to be not working

[hardkid]

The prog says: Movie number could not be extracted from the page (script outdated?). Please update the script. Thanx!

[antp]

This problem comes from time to time.
In this case it is IMDb which blocks bots from time to time (probably when the site has too much load from them).
Nothing can be done except waiting that it is available again.

[duncan]

The other script "IMDB Via API" works fine.

[hardkid]

This problem comes from time to time.
In this case it is IMDb which blocks bots from time to time (probably when the site has too much load from them).
Nothing can be done except waiting that it is available again.

Thank you.

[hardkid]

The other script "IMDB Via API" works fine.

Yes! In my case it was damaged thus not working either. Re-downloaded and now it's OK. Thank you.

[Saephyroth]

In my case the issue is still there. I get "Movie number could not be extracted from the page", regardelss the title I put.
I re-doenloaded the script, it did not solve the issue.
Any clue?

[antp]

Unfortunately since IMDb is blocking bots, there is nothing that can really be done. At least I haven't yet found something that works.
In the meantime, you'll have to use the other script, "IMDB ( via API )".

[Saephyroth]

Unfortunately since IMDb is blocking bots, there is nothing that can really be done. At least I haven't yet found something that works.
In the meantime, you'll have to use the other script, "IMDB ( via API )".

In my case, "IMDB ( via API )" is not working either: I tried both BatchMode 0 and 2 and it is unable to get the following movie info:
https://www.imdb.com/it/title/tt36456563/

[antp]

For me the API script works on that one. What happens for you? Is there a message saying it does not find it? Or just empty data returned?

[Saephyroth]

HTTP/1.1 400 Bad Request

[Saephyroth]

It's working now, using latest IMDB ( via API )

[MrObama2022]

This is how it could be fixed (MAYBE!!!), at least for Windows (this should work on linux too): chromedriver + chrome + batch script + hidden win32 desktop
I did a quick & dirty test and this seems to work.

This is the circus:
- AMC does not use GetPage but it calls an external script using chromedriver (chromedriver is a standalone application to control chrome using its api, you can download chromedriver and chrome from this url)
- chromedriver then calls chrome and ask chrome to open an url, then it output the html back
- the hidden win32 desktop avoid you see chrome when it works, so all the operation is done in "background"/stealth mode

Of course, I can help user to setup the script with all the steps.

Cons:
- slow (!)

This is a lot of work to be done but I'll only develop this if @antp agree this script is official. Too muck work for an unofficial script.

[Dorohedoro]

This is how it could be fixed (MAYBE!!!), at least for Windows (this should work on linux too): chromedriver + chrome + batch script + hidden win32 desktop
I did a quick & dirty test and this seems to work.

This is the circus:
- AMC does not use GetPage but it calls an external script using chromedriver (chromedriver is a standalone application to control chrome using its api, you can download chromedriver and chrome from this url)
- chromedriver then calls chrome and ask chrome to open an url, then it output the html back
- the hidden win32 desktop avoid you see chrome when it works, so all the operation is done in "background"/stealth mode

Of course, I can help user to setup the script with all the steps.

Cons:
- slow (!)

This is a lot of work to be done but I'll only develop this if @antp agree this script is official. Too muck work for an unofficial script.

On the contrary, my humble opinion is that the API version should be the official version, is reliable and fast.

[gerol]

I'm very interested in MrObama2022's suggestion and would incorporate it into the OFDb-mobi-IMDb script, if it works (what does "slow" mean?). I've also considered a similar solution myself, but I can't quite get it to work technically (I'm certain, I can access the required page source code with all common browsers [Chrome, Firefox, Edge], if JavaScript is enabled).

Regarding the API version: We don't know, how long this version will continue to work. Why do IMDb/Amazon block "amateurs" from screen scraping, while allowing "professionals" access to their API via RapidAPI?

It would be great, if there were two working versions.

Best regards,
Gerol

[MrObama2022]

Code: Select all

what does "slow" mean?

In my first test, this is "slow" because it waits all the page is loaded, that is all the javascript and all the images/video. Only after the page is full loaded I can get the html. Maybe it could working faster if I can disable media loading, leaving only js on.

IMHO, this is why solution is legal: it's like a comet browser / ai browser that access a page and give you a summary/report with all details. We are accessing the page using chrome and we are using an "agent" (AMC iMDB script) to resume the details in a PRIVATE (not PUBLIC) storage. I'm not a lawyer (my wife is ...) and this is only my personal opinion. What we are not doing: we are not using a crawler to create a perfect copy of all the site, but we are accessing only to specific page/url; we are not accessing reserved zone (hacking the site); we are not freely accessing to paying premium content; we are not reselling or redistributing their content. We are just using an agent to resume page info for our private reading/usage.

How the script should work:

=== STEP 1 ===
- check if chromedriver exists and if it's up a running
- if it does not exist, it tell you how download and where to install
- if it's not running, the script launch it (it works in background)

=== STEP 2 ===
- it uses chromedriver to access the pages and analyze html

I could create a unit for chromedriver only so other script could reuse it (other site could self protect from bot)

[gerol]

IMHO, this is why solution is legal:

That's my opinion too. I think AMC users used the program to retrieve information about individual (!) films for private purposes and to supplement their personal databases. If the data is freely accessible and no login or registration is circumvented, this should be permissible (at least under German law). Therefore, I can't imagine that we've upset IMDb.

However, it's conceivable that companies are using AI to retrieve the entire (!) IMDb database and present it to their clients. I understand that IMDb doesn't want this (AMC users are simply out of luck). API access might not be problematic for IMDb because AI doesn't interfere with such interfaces.

With the solution you suggested, we use a standard internet browser that IMDb allows access to.

I had been wondering if it would be possible to use the AMC script to access the default Windows internet browser (instead of GetPage), load the desired page, temporarily cache it, and then further examine the temporary file with the script. But I suppose that's not possible?

Best regards,
Gerol

[Dorohedoro]

Since IMDB is blocking scrappers I seriously suggest that this issue should be addressed in private. Since this forum is public, any method suggested in here will be seen by IMDB team, they will block anything you guys make and who know if they decide to take this matter to the next level.

They don't want their data to be scrapped, period.

I reiterate, the API is the way to go.

[gerol]

I seriously suggest that this issue should be addressed in private

I completely agree.

However, I still don't understand where the belief comes from that API access will be tolerated indefinitely (feel free to reply via PM).

Regards,
Gerol