[UPD][EN] MyAnimeList.net v1.1

FinderX · Post by **FinderX** » 2010-05-29 11:38:01

MyAnimeList.net
ver 1.1

FIX: users reviews not appear with accented chars (thk to antp)
FIX: [Related Anime] not appear
FIX: reviews not appear when title have parentheses "()"
FIX: sometime name and tag of fansub have tabs in the names

ADD: the LANGUAGE of the fansub (terrible forgetfulness)
ADD: use the "More Links" feature of PickTree = less connections
ADD: the display of number page in PickTree (page x of y)
ADD: the option Reviews that choose for FULL or SIMPLE
ADD: the option Picture that choose for BIG or SMALL
ADD: const to easy change parameters of PERCENT
ADD: const to easy change parameters of CHAR SEPARATOR
ADD: const to easy change parameters of LENGTH SEPARATOR
ADD: Improved code, controls to keep only the necessary connections

I thank first antp, because he told me where the problem was that I could not see, but my fault is at 99% (lol, he knows)

The option Reviews have two option, FULL means all the reviews in Reviews Page, and SIMPLE only two in Details Page.

The option Reviews and Picture reduce the connections at 2 or 3 (including picture) when are SIMPLE and SMALL respectively. Instead of 4 or 5 (including picture).

Now there are less conections when disable picture and/or comments. In this case is 1 connection needed.

The constants is for an easy change of the percentage of match of the title in bath mode, an easy change of the format on the user separator in Comments and its length.

These found in the first few lines of script, only move to EDITOR mode and you can see them.

DEFAULT_PERCENT
Percentage of match of the title in Bath Mode

USER_CHAR_SEPARATOR
The char with that use for the separator

For example

USER_CHAR_SEPARATOR = '_'

Code: Select all

blablablablablablablablablablabla
_________________________________

User:

blablablablablablablablablablabla
...

USER_CHAR_SEPARATOR = '#'

Code: Select all

blablablablablablablablablablabla

#################################

User:

blablablablablablablablablablabla
...

USER_CHAR_SEPARATOR = '-'

Code: Select all

blablablablablablablablablablabla

---------------------------------

User:

blablablablablablablablablablabla
...

USER_SEPARATOR_LENGTH
The length of the user separator

Is you not like separator put an empty string '' in USER_CHAR_SEPARATOR or 0 in USER_SEPARATOR_LENGTH

Code Next Post

FinderX · Post by **FinderX** » 2010-05-29 11:38:39

Code: Select all


(***************************************************

Ant Movie Catalog importation script
www.antp.be/software/moviecatalog/

[Infos]
Authors=FinderX
Title=MyAnimeList
Description=Script for anime information in MyAnimeList.net
Site=http://myanimelist.net/
Language=EN-Anime
Version=1.1 @ 2010-05-29
Requires=3.5.0
Comments=
License=This program is free software; you can redistribute it and/or modify it under the  terms of the GNU General Public License as published by the Free Software Foundation;  either version 2 of the License, or (at your option) any later version. |
GetInfo=1

[Options]
Batch Mode=1|0|0=No : Normal Mode|1=Yes : Batch Mode
Reviews=0|0|0=Simple : First two from Details Page|1=Full : All from Reviews Page if has more of two
Picture=0|0|0=Small : Thumbnail from Details Page|1=Big : Fullsize from 'pic&pid' Page, if exists or else thumbnail

***************************************************)

program MyAnimeList;

uses StringUtils7552;

const
    DEFAULT_PERCENT = 70;         // BatchMode Percentage Title, 100 = title perfect match
    USER_CHAR_SEPARATOR = '_';    // Char Separator between users in Comments
    USER_SEPARATOR_LENGTH = 120;  // Separator Length between users in Comments
    MY_ANIME_LIST_QUERY = 'http://myanimelist.net/anime.php?q=';
    MY_ANIME_LIST_ADDRESS = 'http://myanimelist.net/anime/';
    FIND_MORE = 'Find More';
    
   
var
    MovieName: string;
    BatchMode, Cancel, BM_MatchFind: Boolean;
    FullReviews, BigPicture: Boolean;
    NumResults, BM_Percent: Integer;

//------------------------------------------------

(*****Modified version of StringUtils1*****)
function RemoveSpacesAndCompact(AText: string): string;
begin
    result := Trim(StringReplace(StringReplace(StringReplace(AText,#9,' '),#10,' '),#13,' '));
    while (Pos('  ',result) > 0) do result := StringReplace(result,'  ',' ');
end;

//------------------------------------------------

function funcHTMLRemoveTags(Str:string) : string;
begin
    HTMLRemoveTags(Str);
    result := Str;
end;

//------------------------------------------------

function funcHTMLDecode(Str:string) : string;
begin
    HTMLDecode(Str);
    result := Str;
end;

//------------------------------------------------

function ReplaceTroubleChars(Str:string) : string;
begin
    result := StringReplace(Str,'…','…');
    result := StringReplace(result,'«','«');
    result := StringReplace(result,'»','»');
    result := StringReplace(result,'‹','‹');
    result := StringReplace(result,'›','›');
    result := StringReplace(result,'‘','‘');
    result := StringReplace(result,'’','’');
    result := StringReplace(result,'“','“');
    result := StringReplace(result,'”','”');
    result := StringReplace(result,'‚',',');
    result := StringReplace(result,'„','„');
    result := StringReplace(result,'&apos;','''');
    result := StringReplace(result,'"','"');
end;

//------------------------------------------------

function ParserText(Text:string ; StartAt:string ; EndAt:string) : string;
begin
    if (Pos(StartAt, Text) > 0) then
      result :=
        FullTrim(
          funcHTMLDecode(
            ReplaceTroubleChars(
              UTF8Decode(
                funcHTMLRemoveTags(
                  TextBetween(Text,StartAt,EndAt)
      )))))
    else
      result := '';
end;

//------------------------------------------------

procedure ExecutePickTree(var Address:string);
begin

    if not BatchMode then begin
    
        if (NumResults > 0) then begin
        
            if PickTreeExec(Address) then begin
                
                if (Address <> FIND_MORE) then begin
                    AnalyzeMoviePage(Address);
                    Cancel := TRUE; // Exit NormalMode
                end;
                
            end else begin
            
                Address := ''; // Cancel Button
                
            end;
            
        end else
        
            ShowInformation('No results found for "' + MovieName + '"' + #13#10 + 'Try another title');
    end;
end;

//------------------------------------------------

procedure AnalyzeForVariuosPages(Address:string);
var
    pagePos, titlePos, page, totalPages: integer;
    strPage, nextAddress, addressPage, titleTag: string;
    
begin
    titleTag := '<title>Anime - MyAnimeList.net</title>';
    strPage := GetPage(Address);
    pagePos := Pos('Pages (',strPage);
    PickTreeClear;
   
    if (pagePos > 0 ) then begin
        addressPage := TextBetween(strPage,'Pages','</div>');
        totalPages := StrToInt(TextBetween(addressPage,'(',')'),0);

        for page := 1 to totalPages do begin
            PickTreeAdd('MyAnimeList - Page ' + IntToStr(page) + ' of ' + IntToStr(totalPages),'');
            AnalyzePage(strPage);
            if (page < totalPages) then PickTreeMoreLink(FIND_MORE);
            ExecutePickTree(Address);
            PickTreeClear;

            if BM_MatchFind or (Address <> FIND_MORE)
                then Break;

            if (page < totalPages) then begin
                nextAddress := TextBetween(addressPage,'[' + IntToStr(page) + '] <a href="?q=','>');
                nextAddress := MY_ANIME_LIST_QUERY + nextAddress;
                strPage := GetPage(UrlEncode(nextAddress));
                addressPage := TextBetween(strPage,'Pages','</div>');
            end;
        end;
        
    end else begin
        
        titlePos := Pos(titleTag,strPage);
        if (titlePos > 0) then begin
            PickTreeAdd('MyAnimeList','');
            AnalyzePage(strPage);
            ExecutePickTree(Address);
        end else begin
            AnalyzeMoviePage(strPage);
            Cancel := TRUE; // Exit NormalMode
        end;
    end;
    
    PickTreeClear;
end;

//------------------------------------------------

procedure AnalyzePage(PageStr:string);
var
    movieAddr, movieTitle, movieID: string;
    titleTag, title: string;
    titlePos, titleLine, compare: Integer;
    strPage: TStringList;
    
begin
    strPage := TStringList.Create;
    strPage.Text := PageStr;
    titleTag := '<a href="' + MY_ANIME_LIST_ADDRESS;
    
    titleLine := FindLine(titleTag,strPage,0);
    
    while (titleLine > -1) do begin
    
        title := strPage.GetString(titleLine);
        titlePos := Pos('<strong>',title);
            
            if (titlePos > 0) then begin
                movieId := TextBetween(title,titleTag,'/');
                movieAddr := MY_ANIME_LIST_ADDRESS + MovieId;
                movieTitle := ParserText(title,'<strong>','</strong>');
                
                if BatchMode then begin
                    compare := CompareWords(movieTitle,movieName);
                    if (BM_Percent <= compare) then begin
                        AnalyzeMoviePage(movieAddr);
                        BM_MatchFind := TRUE;
                        Break;
                    end;
                end;
                
                PickTreeAdd(movieTitle,movieAddr);
                NumResults := NumResults + 1;
            end;
            
        titleLine := FindLine(titleTag,strPage,titleLine + 1);
    end;
    
    strPage.Free;
end;

//------------------------------------------------

procedure AnalyzeMoviePage(Address:string);
var
    page: TStringList;
    lineNr, line, atPos, revLine, numRev: integer;
    reviewsAddress, pictureAddress, titleAlt: string;
    pictureTag, reviewsTag, userTag, groupTag: string;
    fansub, info, group, user, review, reviews: string;
    item, thumbPicture, separator: string;
    
begin
    fansub := '<a href="http://myanimelist.net/fansub-groups.php?id=';
    thumbPicture := 'http://cdn.myanimelist.net/images/anime/';
    pictureTag := '<a href="' + MY_ANIME_LIST_ADDRESS;
    reviewsTag := '<a  href="' + MY_ANIME_LIST_ADDRESS; // the doble space is important
    userTag := '<a href="http://myanimelist.net/profile/';
    page := TStringList.Create;
    
    
    // URL
    item := Copy(Address,1,Length('http://'));
    atPos := Pos('http://',item);
    if (atPos = 1) then begin
        page.Text := GetPage(Address);
        SetField(fieldURL,Address);
    end else begin
        page.Text := Address;
        lineNr := FindLine(MY_ANIME_LIST_ADDRESS,page,0);
        item := MY_ANIME_LIST_ADDRESS + TextBetween(page.GetString(lineNr),MY_ANIME_LIST_ADDRESS,'/');
        SetField(fieldURL,item);
    end;
    
    
    // Original Title - Title
    lineNr := FindLine('<title>',page,0);
    SetField(fieldOriginalTitle,ParserText(page.GetString(lineNr),'<title>',' - MyAnimeList.net</title>'));
    
    
    // Picture Address
    lineNr := FindLine('pic&pid',page,lineNr);
    if (lineNr > -1) and BigPicture then
        pictureAddress := MY_ANIME_LIST_ADDRESS + TextBetween(page.GetString(lineNr),pictureTag,'">')
    else begin
        lineNr := FindLine(thumbPicture,page,0);
        pictureAddress := thumbPicture + TextBetween(page.GetString(lineNr),thumbPicture,'" alt="');
    end;
    
    
    // TranslatedTitle - Alternative Title(s)
    lineNr := FindLine('Alternative Titles',page,lineNr);
    
    titleAlt := ParserText(page.GetString(lineNr),'English:','</div>');
    
    item := ParserText(page.GetString(lineNr),'Synonyms:','</div>');
    if (item <> '') then begin
        if (titleAlt <> '') then titleAlt := titleAlt + ', ';
        titleAlt := titleAlt + item;
    end;
    
    item := ParserText(page.GetString(lineNr),'Japanese:','</div>');
    if (item <> '') then begin
        if (titleAlt <> '') then titleAlt := titleAlt + ', ';
        titleAlt := titleAlt + item;
    end;
    
    SetField(fieldTranslatedTitle,titleAlt);
    
    
    // Director - Type
    lineNr := FindLine('Type:',page,lineNr);
    SetField(fieldDirector,ParserText(page.GetString(lineNr),'Type:','</div>'));
    
    
    // Disks - Episodies
    lineNr := FindLine('Episodes:',page,lineNr);
    item := page.GetString(lineNr) + '¡¿@';
    SetField(fieldDisks,ParserText(item,'Episodes:','¡¿@'));
    
    
    // Year - Year of start aired only
    lineNr := FindLine('Aired:',page,lineNr);
    item := Copy(TextAfter(page.GetString(lineNr),', '),1,4);
    SetField(fieldYear,item);
    
    
    // Productor - Studio
    lineNr := FindLine('Producers:',page,lineNr);
    item := page.GetString(lineNr) + page.GetString(lineNr + 1);
    atPos := Pos('None found',item);
    if (atPos = 0)
        then item := ParserText(item,'Producers:','</div>')
        else item := '';
    SetField(fieldProducer,item);
    
    
    // Source - Genres
    lineNr := FindLine('Genres:',page,lineNr);
    item := page.GetString(lineNr) + page.GetString(lineNr + 1);
    SetField(fieldSource,AnsiMixedCase(ParserText(item,'Genres:','</div>'),' ,-'));
    
    
    // Reviews Address
    lineNr := FindLine(reviewsTag,page,lineNr);
    atPos := Pos('Reviews',page.GetString(LineNr));
    
    while (atPos = 0) do begin
        lineNr := FindLine(reviewsTag,page,lineNr + 1);
        atPos := Pos('Reviews',page.GetString(LineNr));
    end;
    
    reviewsAddress := MY_ANIME_LIST_ADDRESS + TextBetween(page.GetString(lineNr),reviewsTag,'">');
    
    
    // Description - Synopsis
    lineNr := FindLine('<h2>Synopsis</h2>',page,lineNr);
    item := page.GetString(lineNr);
    atPos := Pos('</td>',item);
    while (atPos = 0) do begin
        lineNr := lineNr + 1;
        item := item + page.GetString(lineNr);
        atPos := Pos('</td>',item);
    end;
    
    item := StringReplace(item,'<br />',#13#10);
    SetField(fieldDescription,ParserText(item,'<h2>Synopsis</h2>','</td>'));
    
    
    // Actors - Additional information - Related Anime
    line := FindLine('Related Anime',page,lineNr);
    if (line > -1) then begin
        info := '[Related Anime]' + #13#10;
        item := page.GetString(line) + '¡¿@';
        item := StringReplace(item,'<br>',#13#10);
        info := info + ParserText(item,'Related Anime','¡¿@') + #13#10 + #13#10;
        lineNr := line;
    end;
    
    
    // Number of Reviews, in reviews page
    lineNr := FindLine('More reviews',page,lineNr);
    numRev := StrToInt(TextBetween(page.GetString(lineNr),'More reviews (',')'),0);
    
    
    // Actors - Additional information - Opening Theme
    lineNr := FindLine('Opening Theme',page,lineNr);
    item := page.GetString(lineNr) + '¡¿@';
    atPos := Pos('No opening themes found',item);
    if (atPos = 0) then begin
        info := info + '[Opening Theme]' + #13#10;
        item := ParserText(item,'Opening Theme','¡¿@');
        item := StringReplace(item,'#',#13#10 + '#');
        info := info + FullTrim(item) + #13#10 + #13#10;
    end;
    
    // Actors - Additional information - Ending Theme
    lineNr := FindLine('Ending Theme',page,lineNr);
    item := page.GetString(lineNr) + '¡¿@';
    atPos := Pos('No ending themes found',item);
    if (atPos = 0) then begin
        info := info + '[Ending Theme]' + #13#10;
        item := ParserText(item,'Ending Theme','¡¿@');
        item := StringReplace(item,'#',#13#10 + '#');
        info := info + FullTrim(item) + #13#10 + #13#10;
    end;
    
    // Actors - Additional information - Fansubs
    lineNr := FindLine(fansub,page,lineNr);
    if (lineNr > -1) then
        info := info + '[Fansubbing Groups]' + #13#10;
    
    while (lineNr > -1) do begin
        group := TextBefore(page.GetString(lineNr),'</a>','') + '¡¿@';
        info := info + RemoveSpacesAndCompact(ParserText(group,'>','¡¿@'));
        groupTag := ParserText(page.GetString(lineNr),'<small>[',']</small>');
        if (groupTag <> '') then
            info := info + ' - [' + RemoveSpacesAndCompact(groupTag) + ']';
        groupTag := ParserText(page.GetString(lineNr),'<small>(',')</small>');
        if (groupTag <> '') then
            info := info + ' - (' + RemoveSpacesAndCompact(groupTag) + ')';
        info := info + #13#10;
        lineNr := FindLine(fansub,page,lineNr + 1);
    end;
    
    SetField(fieldActors,FullTrim(info));
    
    
    // User Separator for Comments
    if (USER_CHAR_SEPARATOR <> '') then begin
        separator := StringOfChar(USER_CHAR_SEPARATOR,USER_SEPARATOR_LENGTH);
        if (separator = '') or (USER_CHAR_SEPARATOR = '_')
            then separator := separator + #13#10
            else separator := #13#10 + separator + #13#10;
    end else
        separator := #13#10;
    
        
    // Comments - Reviews
    if CanSetField(fieldComments) then begin
    
        if (numRev > 0) and FullReviews then begin
            // Destroy Details Page
            page.Free;
            
            // Create Reviews Page
            page := TStringList.Create;
            page.Text := GetPage(UrlEncode(reviewsAddress));
        end;
        
        lineNr := FindLine(userTag,page,0);
        
        while (lineNr > -1) do begin
            user := page.GetString(lineNr);
            atPos := Pos('<br />',user);
            
            if (atPos > 0) then begin
                user := ParserText(user,'">','</a>');
                reviews := reviews + user + ':' + #13#10 + #13#10;
                revLine := FindLine('</table>',page,lineNr);
                
                repeat
                    atPos := Pos(#9#9 + '</div>',page.GetString(revLine + 1));
                    
                    if (atPos = 1) then begin
                        
                        repeat
                            review := review + page.GetString(revLine);
                            atPos := Pos('<div>',page.GetString(revLine));
                            revLine := revLine + 1;
                        until (atPos > 0);
                        
                        review := StringReplace(review,'<br />',#13#10);
                        review := StringReplace(review,'read more</a>',' ');
                        review := ParserText(review,'</table>','<div>');
                        reviews := reviews + review + #13#10;
                        reviews := reviews + separator + #13#10;
                        review := '';
                        Break;
                    end;
                    
                    revLine := FindLine('</table>',page,revLine + 1);
                until (revLine = -1);
            
            end;
            
            lineNr := FindLine(userTag,page,lineNr + 1);
        end;
        
        SetField(fieldcomments,FullTrim(reviews));
    end;
    
    
    // Destroy Details or Reviews Page
    page.Free;
    
    
    // Picture
    if CanSetPicture then begin
    
        atPos := Pos(thumbPicture,pictureAddress);
        if (atPos > 0) then
            GetPicture(UrlEncode(pictureAddress))
            
        else begin
            
            // Create Picture Page
            page := TStringList.Create;
            page.Text := GetPage(UrlEncode(pictureAddress));
            pictureTag := '<table id="dialog"';
        
            lineNr := FindLine(pictureTag,page,0);
            pictureAddress := TextBetween(page.GetString(lineNr),'<img src="','">');
            GetPicture(UrlEncode(pictureAddress));
        
            // Destroy Picture Page
            page.Free;
        end;
    end;
end;

//----------------------MAIN-PROGRAM-------------------------

begin
    if (CheckVersion(3,5,0)=FALSE) then begin
        ShowWarning('Required Ant Movie Catalog version 3.5 or higher');
        EXIT;
    end;
    
    BatchMode := GetOption('Batch Mode') = 1;
    FullReviews := GetOption('Reviews') = 1;
    BigPicture := GetOption('Picture') = 1;
    BM_MatchFind := FALSE;
    BM_Percent := DEFAULT_PERCENT;
    MovieName := GetField(fieldOriginalTitle);
    
    if (MovieName = '') then begin
        MovieName := GetField(fieldTranslatedTitle);
        MovieName := TextBefore(MovieName,',','');
    end;
    
    if BatchMode then
        AnalyzeForVariuosPages(MY_ANIME_LIST_QUERY + UrlEncode(MovieName))
        
    else begin // NormalMode
    
        repeat
            if Input('MyAnimeList', 'Search:', MovieName)
                then AnalyzeForVariuosPages(MY_ANIME_LIST_QUERY + UrlEncode(MovieName))
                else Cancel := TRUE; // Exit NormalMode
        until Cancel;
        
    end;
end.

Post by **antp** » 2010-05-29 15:45:55

Thanks

FinderX · Post by **FinderX** » 2010-05-30 03:22:39

when update from [auto-update] is not appear, is because the first line is blank, please you can delete the first line

THANKS

FinderX · Post by **FinderX** » 2010-05-30 03:32:40

Last minute tweaks...

1) I ate a quote...

before

Code: Select all

ShowInformation('No results found for "' + MovieName + #13#10 + 'Try another title');

Now

Code: Select all

ShowInformation('No results found for "' + MovieName + '"' + #13#10 + 'Try another title');

2) I add erroneously an UTF8Decode in line 195...

before

Code: Select all

PickTreeAdd(UTF8Decode(movieTitle),movieAddr);

Now

Code: Select all

PickTreeAdd(movieTitle,movieAddr);

Is perfect now, sorry I'm human...

Post by **antp** » 2010-05-30 20:52:36

Sorry for the error, I made the changes