hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "S.L" <simpleliving...@gmail.com>
Subject Re: Error trying to download a Gzip file.
Date Tue, 02 Apr 2013 21:28:21 GMT
Thanks Stephen .I completely agree with you , I am just accessing publicly
available data , one thing that I would like to mention though is that I
get the following warning every time this happens .

*Invalid cookie header: "Set-Cookie: ". Cookie name may not be empty*.

Even though I have  set a lenient cookie -policy as illustrated here in
section 3.5 .

http://hc.apache.org/httpcomponents-client-ga/tutorial/html/statemgmt.html#d4e777






On Tue, Apr 2, 2013 at 4:19 PM, Stephen J. Butler
<stephen.butler@gmail.com>wrote:

> This really doesn't sound like a problem with HttpClient. Rather, an issue
> with Walmart's servers. Maybe you're querying the server too often and
> they're rate limiting you. Or if the same pattern always works in a browser
> then they're sniffing your UserAgent and doing something different.
>
> In any event, if this is an approved API for Walmart.com, you need to
> contact them. Or just cache the response yourself in the code and use a
> cached value when you get a 404. There's nothing wrong with HttpClient.
> It's doing exactly what the server told it to do.
>
>
>
> On Tue, Apr 2, 2013 at 3:12 PM, S.L <simpleliving016@gmail.com> wrote:
>
> > Stephen,
> >
> > Thanks, I now checked the reason phrase and it returns "Not Found" when I
> > get a non gzip page not found error and an OK when I successfully get a
> > gzip file download. As I said earlier in multiple iterations , the first
> > one is successful almost  all the time and any subsequent request after
> > that can fail. Is there any way I can "stick" to the fist successful
> server
> > instance using the HttpClient , is there any way around  ? Thanks for
> your
> > help.
> >
> >
> > On Mon, Apr 1, 2013 at 11:48 PM, Stephen J. Butler <
> > stephen.butler@gmail.com
> > > wrote:
> >
> > > I notice you aren't checking your status code. Error response have a
> > > body/response entity too. Take a look at
> > > response.getStatusLine().getStatusCode() and getStatusReasonPhrase(). I
> > bet
> > > the server is limiting you in the instances where you're seeing
> > non-gziped
> > > content.
> > >
> > >
> > > On Mon, Apr 1, 2013 at 10:42 PM, S.L <simpleliving016@gmail.com>
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > I use the following code and run the method multiple times, a few
> > times I
> > > > get a response in gzip which is what I expect and a few time I get a
> > > > response that is completely different(non Gzip and html format)
> > .However
> > > if
> > > > I download the same URL multiple times using Mozilla or IE I
> > consistently
> > > > get the same GZIP response , Is this an error with the server I am
> > trying
> > > > to reach to , or do I need to set parameters to get a consistent
> > > response ?
> > > >
> > > > Thanks.
> > > >
> > > > The URL I am trying to download is *
> > > > http://www.walmart.com/navigation6.xml.gz* , can you please let me
> > know
> > > ?
> > > > Thanks
> > > >
> > > >
> > > >
> > > >
> > > >   public static byte[] dowloadURL(URL urlToDownload) {
> > > >
> > > >         InputStream iStream = null;
> > > >         byte[] urlBytes = null;
> > > >
> > > >         try {
> > > >
> > > >             //HttpClient httpClient = new HttpClient();
> > > >             org.apache.http.client.
> > > > HttpClient httpClient = new DefaultHttpClient();
> > > >
> > > >
> > > >             HttpGet httpget = new HttpGet(urlToDownload.toString());
> > > >
> > > >             HttpResponse response  = httpClient.execute(httpget);
> > > >
> > > >             iStream = response.getEntity().getContent();
> > > >
> > > >             urlBytes = IOUtils.toByteArray(iStream);
> > > >             String responseString = new String(urlBytes);
> > > >             System.out.println(" >>> The response  string for
"
> > > > +urlToDownload.toString()+  " is "   +responseString);
> > > >
> > > >         } catch (IOException e) {
> > > >             System.err.printf("Failed while reading bytes from %s:
> %s",
> > > >                     urlToDownload.toExternalForm(), e.getMessage());
> > > >             e.printStackTrace();
> > > >             // Perform any other exception handling that's
> appropriate.
> > > >         } finally {
> > > >             if (iStream != null) {
> > > >                 try {
> > > >                     iStream.close();
> > > >                 } catch (IOException e) {
> > > >                     // TODO Auto-generated catch block
> > > >                     e.printStackTrace();
> > > >                 }
> > > >             }
> > > >         }
> > > >
> > > >         return urlBytes;
> > > >     }
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message