SetCookies()/AddCookie()/GetCookies() not working

[yeti]

During my tests with the Google "Bad Request" response, I figured out that Google needs acceptance to use cookies. To do this, I tried to use SetCookie() or GetPage3() to add this needed cookie. But the functions don't work.

Hint: Workaround to use Google without cookie acceptance: Use http:... instead of https:... But don't now, how long this work.

To test cookies, try:

Code: Select all

procedure Test();
var
  Page, Address: string;

begin
  Address := 'https://www.google.de/search?q=allintext%3A+Holy+Motors+%22Inhaltsangabe+%26+Details%22+site%3AFilmstarts.de/kritiken/';
  SetCookies('TESTCOOKIE=ANTTest', 'www.google.de');
  ShowMessage('Cookies: "' + GetCookies('www.google.de') + '"');
  Page := GetPage(Address);  // Bad Request
  //Page := GetPage3(Address, '', 'TESTCOOKIE=ANTTest'); //<-- no cookie send, too
end;

ShowMessage() shows an empty String. No Cookie.

And my Proxy shows as the first request (in later requests a cookie is send, because google set it in the first response. But this is not my added cookie. To clean the cache, close and reopen the scripting-window)

Code: Select all

GET https://www.google.de/search?q=allintext%3A+Holy+Motors+%22Inhaltsangabe+%26+Details%22+site%3AFilmstarts.de/kritiken/ HTTP/1.1
Content-Type: */*
Accept: text/html, */*
User-Agent: Mozilla/5.0 (compatible; Ant Movie Catalog)
Authorization: Basic Og==
Proxy-Authorization: Basic Og==
Host: www.google.de

No Cookie set. I've expected:

Code: Select all

GET https://www.google.de/search?q=allintext%3A+Holy+Motors+%22Inhaltsangabe+%26+Details%22+site%3AFilmstarts.de/kritiken/ HTTP/1.1
Content-Type: */*
Accept: text/html, */*
User-Agent: Mozilla/5.0 (compatible; Ant Movie Catalog)
Authorization: Basic Og==
Proxy-Authorization: Basic Og==
Host: www.google.de
Cookie: TESTCOOKIE=ANTTest

Whats wrong?

Greetz,
yeti

[antp]

There were also other problems causing Bad Request error.
You may try with the beta/version test that I posted there: viewtopic.php?p=59886#p59886
(see the first post of that thread for more info)

But it is possible that there is a problem with the cookies. The site moviemeter also has such problem with a Bad Request error after the first request. It may be related.
But I don't have time to investigate that now, I'll try to check that later.

[yeti]

Ok, with 4.2.3.0 and GetPage4(Address, '', 'TESTCOOKIE=ANTTest', ''), I still get a 302 (moved page), and now the moved page is a 200 (ok), not a 400 (bad request). The 302 response contains a Set-Cookie "CONSENT=PENDING+xxx". And the moved page contains the cookie consent request that i need to confirm.

My TESTCOOKIE is still not transmitted, so I can't transmit the needed "CONSENT=YES+..."-Cookie to google. If I inject the CONSENT-cookie in the request, the search result is returned.

GetCookies() can't work, because the code in getscript.pas is commented out. SetCookies()/AddCookie() not. So it should work, but it does not Or I'm doing something wrong.

So currently only http works instead of https, or changing the user agent. Not really satisfying.

[antp]

I don't know how this cookie management is supposed to work, it was added by Mickaël and I just did a few changes when upgrading to a newer version of Indy for better HTTPS support.
I don't remember if these functions were commented at the time of that upgrade because there was a problem, or if that happened earlier.
I guess that the few scripts that use cookies are just setting them and do not really rely on what the site sends?