hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kwik, Micky" <micky.k...@sogeti.nl>
Subject RE: HttpClient returns different response compared to browser
Date Tue, 20 May 2008 14:20:16 GMT
Hey Sebb,
 
Thnx alot. I used a HTTP sniffer and found out that the requests from my client were not correct.
Problem is solved now :)
 
Micky

________________________________

Van: sebb [mailto:sebbaz@gmail.com]
Verzonden: di 20-5-2008 15:22
Aan: HttpClient User Discussion
Onderwerp: Re: HttpClient returns different response compared to browser



On 20/05/2008, Kwik, Micky <micky.kwik@sogeti.nl> wrote:
> Hi,
>
>  I wrote a simple client to fetch documents from some websites. But I found that the
HttpClient often gets a different response compared to the browser even if the HTTP status
code is 200.

In what way is the response different?

> For example this URL: http://www.elsevierfiscaal.nl/els/enc/productserviceoverzicht/id1101-31813/search/true/channelId/1101/update-14-aangifte-assistent-2008.html
<https://webmail.sogeti.nl/exchweb/bin/redir.asp?URL=http://www.elsevierfiscaal.nl/els/enc/productserviceoverzicht/id1101-31813/search/true/channelId/1101/update-14-aangifte-assistent-2008.html>
 or http://www.belastingdienst.nl/zakelijk/nieuwsbrief/nieuwsberichten/2008-04-02-08_franke.html
<https://webmail.sogeti.nl/exchweb/bin/redir.asp?URL=http://www.belastingdienst.nl/zakelijk/nieuwsbrief/nieuwsberichten/2008-04-02-08_franke.html>
>
>  Here is my code snippet:
>   HttpClient client = setUpClient(aUrl);
>   GetMethod method = new GetMethod();
>   method.getParams().setParameter("http.useragent", "Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14");
>   method.getParams().setBooleanParameter("http.protocol.single-cookie-header", true);
>   method.setFollowRedirects(false);
>   method.getParams().setCookiePolicy(CookiePolicy.BROWSER_COMPATIBILITY);
>
>   int returnCode = client.executeMethod(method);
>   if (returnCode == HttpStatus.SC_OK) {
>    fetchBody(aUrl, method);
>   }
>  I already varied Cookie policies and followRedirects but with no success. Is there a
way of solving this or is it a case of "http client is not a browser" ?

So long as HttpClient is set up to send the correct HTTP requests, the
server cannot tell if it is talking to a browser or not. [Well, I
suppose it could do some clever tricks with timing or Javascript. But
that is unlikely to be the case here.]

So you just need to find out what the difference is between what the
browser sends and what HttpClient is sending. There may be some extra
hidden fields or other parameters that have been overlooked.

A protocol sniffer such as Wireshark - or a recording proxy - would be
helpful here.

>  Kind regards,
>  Micky
>
>  Disclaimer:
>  This message contains information that may be privileged or confidential and is the
property of Sogeti Nederland B.V. or its Group members. It is intended only for the person
to whom it is addressed. If you are not the intended recipient, you are not authorized to
read, print, retain, copy, disseminate, distribute, or use this message or any part thereof.
If you receive this message in error, please notify the sender immediately and delete all
copies of this message.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org




Disclaimer:
This message contains information that may be privileged or confidential and is the property
of Sogeti Nederland B.V. or its Group members. It is intended only for the person to whom
it is addressed. If you are not the intended recipient, you are not authorized to read, print,
retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive
this message in error, please notify the sender immediately and delete all copies of this
message.


Mime
View raw message