hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Roepcke <...@roepcke.com>
Subject Re: Making ContentLengthInputStream stop when I want, is this safe?
Date Sun, 03 Jul 2005 01:47:08 GMT
On 2-Jul-05, at 4:56 PM, Michael Becke wrote:

> Hi Jim,
> HttpMethod.abort() is probably what you're looking for.
> Your workaround solution could potentially cause some issues.  In
> particular it will be a problem if you end up reusing a connection
> that has been skipped.  The remaining content will still be on the
> wire and it will be read the next time the connection is used.
> Reading to the end of the response is specifically done so that
> connections can be reused.  In your case it sounds like it would be
> better to abort the method, which causes the connection to be closed
> and not reused.

Hi Michael,

Thanks for the advice. :-)  That's interesting that the content would  
still be on the wire.  I guess that's a part of HttpClient or sockets  
I don't understand well enough -- I assumed that when the socket was  
closed the remaining bytes simply "could not get through", and that  
after exhaustInputStream(), the socket would soon after be closed.

I did try get.abort() before devising the workaround, but after  
calling abort, subsequent HTTP methods weren't being executed.  ... I  
just tried it again.  It works a few times but then eventually HTTP  
methods just don't execute.  My guess is, since the connections  
aren't (as far as I can tell) being released to the pool when you  
call abort(), eventually the MultiThreadedHttpConnectionManager's  
connection pool is starved and it has nothing to offer the HttpClient.

Looking at abort(), it calls close() on the connection, but not  
releaseConnection(), which would in turn call  
httpConnectionManager.releaseConnection(this).  I assume this is why  
you say it won't be re-used?  On a quick inspection, I don't see the  
MultiThreadedHttpConnectionManager being notified that the connection  
is finished or unusable.  Do I have to either tell it somehow or  
create a new connection manager?  It doesn't appear abort() does  
anything to tell the connection manager the connection is now unusable.

Here's how I have the read loop when I use abort():

      while ((bytesRead = downloadStream.read(downloadBuffer, 0,  
download_stream_buf_size)) > 0) {
          raf.write(downloadBuffer, 0, bytesRead);
          totalBytesRead += bytesRead;
          if (shouldStop()) {
              get = null
      if (get != null) {
          get = null;
      // [snip]

Do you seen anything wrong with that? (Assuming abort() is working  

I appreciate your advice, hopefully I can get this working properly  
using abort(), because it sounds like that is the intended device for  
my needs.

If you have any suggestions for improvements to abort, I'm very happy  
to try them here.

Thanks again,


> On 7/2/05, Jim Roepcke <jim@roepcke.com> wrote:
>> Hi,
>> I'm writing an application where the user must be able to interrupt
>> and resume downloads at will.
>> I was noticing noticeable pauses when I called releaseConnection() on
>> the GetMethod that I was reading the response from.  Recently I
>> realized this is because deep in HttpClient, releaseConnection
>> eventually calls exhaustInputStream to read and discard the rest of
>> the response (!!!) before closing the connection.  Since the file
>> downloads the user will interrupt are potentially many gigabytes in
>> size, having releaseConnection block while it reads and discards many
>> gigabytes of data is obviously very bad.
>> I created a workaround.  It seems to be working for me, ie:
>> releaseConnection is no longer pausing, but I am guessing the
>> developers of HttpClient had something important in mind when they
>> decided to FORCE responses to be COMPLETELY read.  Could you please
>> let me know if my hack is going to cause bad side-effects?
>> Background info: I'm using the MultiThreadedHttpConnectionManager.
>> At any time I could have up to one (significant) GET download, one
>> (significant) PUT upload, and possibly a very small number of very
>> short DELETE, HEAD and GET requests executing.
>> The hack:
>> In ContentLengthInputStream, I added a method forceSkipTheRest().  It
>> does the following:
>>      public void forceSkipTheRest() {
>>          this.contentLength = this.pos;
>>      }
>> In AutoCloseInputStream, I added an accessor method to get the
>> wrapped [ContentLength]InputStream, so that I can call the
>> forceSkipTheRest method on it.
>> Here is a snippet of code from my Download worker code (which runs in
>> a separate thread managed by Doug Lea's util.concurrent's
>> QueuedExectutor class).  Please refer to the section near the bottom
>> inside the if (shouldStop()) block for the use of the gross hack...
>> // [snip]
>> client.executeMethod(hc, get);
>> int status_code_get = get.getStatusCode();
>> if ((status_code_get / 100) == 2) { // in the 200 series of responses
>>      responseStream = get.getResponseBodyAsStream();
>>      int download_stream_buf_size = 4096;
>>      // TODO: review again if this bufferedinputstream is needed
>>      // it seems it is redundant, the responseStream is probably good
>> enough!!
>>      downloadStream = new BufferedInputStream(responseStream,
>> download_stream_buf_size);
>>      byte[] downloadBuffer = new byte[download_stream_buf_size];
>>      int bytesRead = 0;
>>      raf = new RandomAccessFile(downloadFile, "rw");
>>      while ((bytesRead = downloadStream.read(downloadBuffer, 0,
>> download_stream_buf_size)) > 0) {
>>          raf.write(downloadBuffer, 0, bytesRead);
>>          totalBytesRead += bytesRead;
>>          if (shouldStop()) {
>>              if (responseStream instanceof AutoCloseInputStream) {
>>                  InputStream stream = ((AutoCloseInputStream)
>> responseStream).getWrappedInputStream();
>>                  if (stream instanceof ContentLengthInputStream)
>>                      ((ContentLengthInputStream)
>> stream).forceSkipTheRest();
>>              }
>>              break;
>>          }
>>      }
>>      get.releaseConnection();
>>      get = null;
>>      // [snip]
>> In some quick testing, it seems this is working just fine... however,
>> I wonder if any bad side effects will result from this.
>> I'm also wondering if perhaps I should simply set a flag in
>> ContentLengthInputStream, and if that flag is true, when close() is
>> called on ContentLegnthInputStream it calls forceSkipTheRest before
>> calling exhaustInputStream (so that this interruption works the way I
>> need in all cases, not just in my read loop).
>> Your advice is *greatly* appreciated!!
>> Thanks,
>> Jim

To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org

View raw message