Incomplete Information using IMDB Script

[AllanH]

I'm using the IMDB Script (Import data & picture) for this TV Mini Series,
https://www.imdb.com/title/tt0096535
After selecting it from the Search Results, the Director, Producer, Writer, Composer, and Year do not display anything.
This information is on the Full Cast & Crew page,
https://www.imdb.com/title/tt0096535/fullcredits
Does the Script need to be modified for the TV Mini Series?

[antp]

It is possible that the IMDb script is not currently updated for TV series, as I usually test only movies.
I should try to fix that, but I don't have often time for it. And I don't know if it is an easy fix or if it would require a lot of work (in this latter case, it is unlikely that I can do it).

[AllanH]

I would like to add that the TV Episode, such as https://www.imdb.com/title/tt0213826, and the TV Movie, such as https://www.imdb.com/title/tt0282100, do not have the same problem.
Thanks for your reply.

[MrObama2022]

I think the main problem is translated titles. We can't get all titles, I don't think we can use ajax and I have no clue (I have only bad ideas ...)

[MrObama2022]

By the way, this is my bad idea to get all translated titles, using a free account on rapidapi [ https://rapidapi.com/apidojo/api/imdb8 ]:

USER:
- create a free account on rapidapi and add the his apikey as a parameter in iMDB

EXTERNAL PUBLIC RESOURCE:
- we put up a public gateway, that is https://www.antp.be/gateway.php?apikey= ... on=$action (we can use another host)

iMDB SCRIPT:
- it does not scrape iMDB but it use public gateway

FAQ:
- Why public gateway and not direct query? Because we can't set additional headers in GET request so we have to use a "gateway"/external script
- What is user does not create a rapidapi account? It will get only few alternative titles (the ones we can scrape with no effort in /reference or /releaseinfo urls)

I still have to test rapidapi and that's a lot of work, I have no time but just now I don't have better ideas.

[antp]

That API may indeed be a good solution, more reliable.
If we just need additional parameters for the GET request, maybe it is easier to add that in AMC than making a gateway.

[MrObama2022]

I just spent 5 minutes on this. I sign up on rapidapi, selected personal project in registration form and subscribed free plan. It got 1 minute and rapidapi didn't ask for credit card or other payment method

I used their php code for a test and it worked like a charm, very fast response. Response is in json format (as you can see on their site clicking on "example responses") so I think this is a good solution. To get the requests under 500 / month we can use a mixed approach: "standard" scraping for all fields and api scraping for further fields. If a user does not have a rapidapi key he can't access to further fields.

Or we can switch to full api approach. But if we choose this way, we should also support multiple accounts / rapidapi keys (then any user should subscribe with different emails for multiple accounts) so the script can select a random rapidapi key to perform the query because 500 requests / month are too few.

For my ImdbPas.pas I think I'll stay with the mixed approach so I'll try to use the api only for full alternative titles. But I still have to check if the new iMDB site added further limits to my script.

Let me know if you can add headers to AMC requests.
Note: consider the gateway version could perform also a free "cache" response. That is, user could set a max old response (example: max 5 days old) and then the gateway / cache could check if the movie id is present and the update time is inside user preference. If it's inside the user receive the response from the cache, if it's not inside or the id is not present a new request is performed. It could add also further function as a request counter for each account. But a gateway version is slower then a direct access to rapidapi and speed is crucial.

[antp]

To get the requests under 500 / month we can use a mixed approach: "standard" scraping for all fields and api scraping for further fields. If a user does not have a rapidapi key he can't access to further fields.

Or we can switch to full api approach.

Is the request per field or per movie/search?
If it is per movie, I'd rather have a separate script, so the user can chose if he wants the API (more complete, more reliable on long term) or the traditional script (not requiring signup). As a mixed script has advantage of both but also drawbacks of both, especially risk of breaking at any change on IMDb side.
edit: I see the details of the request on their site, as you don't get all movie info via a single request indeed maybe a mixed script is better by default, so the API will only be used for fields we can't get from the site.
A full API script can still be useful for people who only import only a few movies per month, or for people who want to pay for it

But if we choose this way, we should also support multiple accounts / rapidapi keys (then any user should subscribe with different emails for multiple accounts) so the script can select a random rapidapi key to perform the query because 500 requests / month are too few.

I would not risk that, it is a risk to have AMC banned from that service by helping users to go around these limits.

Let me know if you can add headers to AMC requests.

What is needed actually? Just adding headers X-RapidAPI-Key and X-RapidAPI-Host with associated values? That should be easy, I'll try to do that.

Note: consider the gateway version could perform also a free "cache" response. That is, user could set a max old response (example: max 5 days old) and then the gateway / cache could check if the movie id is present and the update time is inside user preference. If it's inside the user receive the response from the cache, if it's not inside or the id is not present a new request is performed. It could add also further function as a request counter for each account. But a gateway version is slower then a direct access to rapidapi and speed is crucial.

Like for the "multiple account" idea above, I would not risk that either. And that's a lot of work for my side
I would think that 500 request is enough for most of the people.

[antp]

Let me know if you can add headers to AMC requests.

What is needed actually? Just adding headers X-RapidAPI-Key and X-RapidAPI-Host with associated values? That should be easy, I'll try to do that.

On my to-do list today was "make AMC release with the 4.2.3 beta version that is pending since a long time", that will wait a little longer, I made another beta version where I added a GetPage5 method.
Here is a version for it:
https://update.antp.be/amc/beta/amc4232b.rar

Sample use:

Code: Select all

program NewScript;
var
  response: string;
  headers: string;
begin
  headers := 'X-RapidAPI-Key=*userkey*'#13#10'X-RapidAPI-Host=imdb8.p.rapidapi.com';
  response := GetPage5('http://ks3c.antp.be/showheaders.php', '', '', '*/*', headers);
  ShowMessage(response);
end.

(I made a page that returns the headers received, it was easier to test for me )
GetPage5 is like GetPage4 with an extra param, the list of headers using Delphi's TStrings format: a list of "name=value" separated by line breaks.
Here I passed the default values for params 2, 3 and 4.

At the same time, I upgraded to the latest version of MediaInfo

edit: and I also added a PostPage3 function with a similar extra param, just in case it is needed some day.

[MrObama2022]

Thank you, wonderful. Now all should be ease. I just must find more time to develop the patch to my files

[MrObama2022]

I tested and all works fine now but now I have to fix charset translation

[antp]

What do you mean by that?

[MrObama2022]

I think API response uses UTF16 Charset, I'm not sure ...
I'm at work on this ... I would not write a UTF16Decode() function ...

[antp]

That is strange, it is not so much used.
Maybe I can easily add that if needed, I would have to check.

[Guest_2691]

Hello, I was also having the same problem: script not getting the translated titles of the movies. I tried changing the code many times without success (I know almost nothing about coding). Then I had an insight: As we know IMDB automatically translates the titles of the movies according to user country that is accessing the site, then it's just a matter of getting that information. First I disabled all the code in the AKA Name section, then I copied part of the code from a disabled section of the script starting at line 166 (Original Title & Year), made some changes and pasted it into the AKA Name section. I tested it and bingo! It worked. Afterwards I used a VPN for testing some countries like France, Germany and Hungary. The script returned the translated titles correctly for each one. I'm posting the code below so you can change it if needed (maybe it has useless stuff).

Code: Select all

if CanSetField(fieldTranslatedTitle) then
  begin
    originalTitle := TextBefore(PageText, '(original title)', '</h1>');
    HTMLRemoveTags(originalTitle);
    HTMLDecode(originalTitle);
    originalTitle := RemoveSpaces(originalTitle, true);
    if (originalTitle <> '') and CanSetField(fieldTranslatedTitle) then
    begin
      SetField(fieldTranslatedTitle, FullTrim(originalTitle));
    end;
    Value := TextBetween(PageText, '<title>', '</title>');
    p := Pos(' (1', Value);
    if p = 0 then
      p := Pos(' (2', Value);
    if p = 0 then
      p := Pos(' (TV', Value);
    if p > 0 then
    begin
      Value2 := Copy(Value, 0, p-1);
      HTMLDecode(Value2);
      if (originalTitle = '') and CanSetField(fieldTranslatedTitle) then
        SetField(fieldTranslatedTitle, FullTrim(Value2));
    end;
  end;
  

[antp]

Thanks, I'll check that in the next days / next week to include it in the main script

[antp]

I included your change as an option ("TranslatedTitle" to set to 1, by default it is set to 2 for the old behaviour) but I can't really test it - I don't know why IMDb does not show me alternative/translated titles on the main page, though that it was doing that before
I also included a fix for title search result, to add missing spaces - I thought I already did that previously, but it seems not
(version 4.023 via "update scripts")

[Guest_2691]

Hello, first time I run the new script (version 4.023) TranslatedTitle was set to 2 by default. This old code doesn't work. I set it to 1 and it worked fine. Again I used VPN to test other countries like Austria, Slovenia, Vietnam and it worked as well. I think there's an issue related to Belgium, the country you live in. Most of the movies I've searched for, Belgium isn't listed on IMDB aka page. For few movies there is 2 translated titles for Belgium. The movie "The Jewel of the Nile" for example, there's only one Belgium translated title in Flemish. Even tough the script didn't get it (VPN was set to Belgium). I think it's because Belgium has three official languages and IMDB has no support for it.
My conclusion: the new code is a workaround that will work in most cases but not for all.

[Dorohedoro]

I'm having issues with the awards, the script is not taking that field of information. I'm using the lastest version of the script.

Thanks!

[Guest_2691]

I'm having issues with the awards, the script is not taking that field of information. I'm using the lastest version of the script.

Thanks!

Here the script is working for Awards. In Script Options you must set Awards to 1 or 2 according to your preference. In fact, Awards information is put in the field Description after summary or in the field Comments.