Mailing-List: contact httpclient-users-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpClient User Discussion" <httpclient-users@hc.apache.org>
Received-SPF: pass (athena.apache.org: domain of stephen.butler@gmail.com
 designates 209.85.214.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAFakBwiH2Ju5pHKpODNZ93yGy+V_BP1-GfSKHY4SnXP7ZY8nGA@mail.gmail.com>
References: 
 <CAFakBwjWsVXGirh_RKPWDZOHPd7=mHnmLdtKMstOfV4i9RRMuQ@mail.gmail.com>
	<CAD4ANxUpUqQC34_pMbMct5th=japbJ=D_DCQxCA8HhGymGxc_Q@mail.gmail.com>
	<CAFakBwgHwoyZjKnsy1EVxdzJvKabgCy679vhEXynUaGccybVyA@mail.gmail.com>
	<CAD4ANxVPhSAM-A8=xMd_wWJQ2Qh6GWWYh2AyocoRVeA-aunNzw@mail.gmail.com>
	<CAFakBwiH2Ju5pHKpODNZ93yGy+V_BP1-GfSKHY4SnXP7ZY8nGA@mail.gmail.com>
Date: Tue, 7 May 2013 10:10:56 -0500
Message-ID: 
 <CAD4ANxVkSLDPg8U0yGGmEu9+xKFr3oua2Q88psS7tPVyno8NGQ@mail.gmail.com>
Subject: Re: why are so different the response headers as reported by wget and
 httpclient?
From: "Stephen J. Butler" <stephen.butler@gmail.com>
To: HttpClient User Discussion <httpclient-users@hc.apache.org>
Content-Type: multipart/alternative; boundary=089e013d0758624b5704dc2239ac

--089e013d0758624b5704dc2239ac
Content-Type: text/plain; charset=ISO-8859-1

On Tue, May 7, 2013 at 3:36 AM, Albretch Mueller <lbrtchx@gmail.com> wrote:

> > It's passing the wrong 'Host' line. That's why you are getting a 404.
>
>  well, the code section in which I set the host I pasted bellow (
> notice how I set the Host as part of the Request Headers  {"Host",
> (httpGet.getURI()).getHost()} ):


Yes, but that's not correct after the redirect. First time it connects the
'Host' line should be 'download.ted.com'. But the second time, after
following the redirect, it needs to be 'video.ted.com'.


> > Are you using standard code to handle the redirect, or writing your own?
>
>  I will have to do the redirect myself and I would like to handle/get
> all the response headers of every redirect. How do you do that? Could
> you point me to some basic redirect code example?
>
> ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
>   HttpGet httpGet = new HttpGet(aGetURL);
>   String aRqLn = httpGet.getRequestLine().toString();   // Request Line
> System.err.println("// __ httpGet.getRequestLine(): |" + aRqLn + "|");
>
> ...
> // __ parsing host from URL
>   String aHost = (httpGet.getURI()).getHost();
> System.err.println("// __ aHost: |" + aHost + "|");
>
>   String[][] aRqHdrs = new String[][]{
>      {"Host", aHost}
>    , {"Connection", "keep-alive"}
>    , {"User-Agent", "Mozilla/5.0 (X11; Linux i686; rv:10.0.4)
> Gecko/20100101 Firefox/10.0.4 Iceweasel/10.0.4"}
>    , {"Accept", "text/html, text/*;q=0.9, image/jpeg;q=0.9,
> image/png;q=0.9, image/*;q=0.9, */*;q=0.8"}
>    , {"Accept-Encoding", "gzip, deflate, x-gzip, x-deflate"}
>    , {"Accept-Charset", "utf-8,*;q=0.5"}
>    , {"Accept-Language", "en-US,en;q=0.9"}
>   };
>

I see, you're setting the request header manually! That's why HttpClient is
failing to properly set the Host header.

I'd suggest you not set the Host header manually and let HttpClient do it.
It will add it itself; that's mandated by the HTTP 1.1 spec.

If you insist on handling the redirect manually, set
the ClientPNames.HANDLE_REDIRECTS parameter to Boolean.FALSE. Then the
return from executing your request will be the raw redirect, and you'll
have to catch that case and resubmit the request on the proper URL.

--089e013d0758624b5704dc2239ac--