hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "St Jacques, Robert" <RStJacq...@crt.xerox.com>
Subject RE: streaming responses
Date Wed, 29 Sep 2004 19:17:43 GMT
Oleg,

This is exactly what I tried to do, but the HttpClient seems to buffer the
response anyway.  I poked through the code a bit, and did some searching on
the mailing list, and this seems to be the case.  Also, when I run tests,
the log output from the HttpClient seems to indicate that the request is
being buffered before I even get an opportunity to call
"getResponseBodyAsStream".  I tried this on a request that is about 17 MB
and just sat back and watched as the response was buffered; I ended up
terminating the program after a minute or two without ever actually getting
past the execute method call.  Here is the output from the logs:

  enter HttpMethodBase.processResponseHeaders(HttpState, HttpConnection)
  enter GetMethod.readResponseBody(HttpState, HttpConnection)
  enter HttpMethodBase.readResponseBody(HttpState, HttpConnection)
  enter HttpMethodBase.readResponseBody(HttpState, HttpConnection)
  enter HttpConnection.getResponseInputStream()
  Buffering response body
<lots of logs detailing the bytes read>

>From here it looks like the "processResponseHeaders" method (which is called
by the HttpMethodBase during an execute) automatically buffers the response.
Is this true, or am I totally misreading the signs?

Thanks,
Bob

-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org] 
Sent: Wednesday, September 29, 2004 3:04 PM
To: Commons HttpClient Project
Subject: Re: streaming responses


Bob,

There's no special magic involved. Make sure you use
HttpMethod#getResponseBodyAsStream, which will return the raw input stream,
and not its buffering counterparts HttpMethod#getResponseBodyAsString and
HttpMethod#getResponseBody.

You should not worry about chunking. HttpClient will decode chunked input on
fly, if necessary. Just grab the raw input stream and do whatever response
retrieval suits your application best.

Hope this helps

Oleg


On Wed, 2004-09-29 at 20:43, St Jacques, Robert wrote:
> Howdy,
> 
> I've just started using HttpClient for testing.  The product that I am 
> developing includes a software download feature that downloads 
> (sometimes very large) files over the internet using HTTP.  The reason 
> we use HTTP is that many of our customers are unwilling to open their 
> proxies or firewalls to other protocols.
> 
> In order to accommodate these downloads, we must either be able to 
> stream the response to a GET, or we need to be able to insure that the 
> response we get from the server will be chunked in such a way that the 
> chunks are small enough to buffer.  I know that HttpClient supports 
> chunking, but I am unable to 'force' our web server to chunk up the 
> data.  Furthermore, the HTTP 1.1 RFC indicates that the server does 
> not need to guarantee that a response, even if chunked, will be small 
> enough to conveniently buffer on the client.
> 
> What is the best way to use HttpClient to retrieve large amounts of 
> data (as large as 200 MB or more)?
> 
> Thanks,
> Bob St. Jacques
> Member Research Technical Staff
> Xerox Corp.
> (585)231-8306
> 


---------------------------------------------------------------------
To unsubscribe, e-mail:
commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail:
commons-httpclient-dev-help@jakarta.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org


Mime
View raw message