hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Claassen <mclaas...@ocie.net>
Subject RE: have question about buffer size
Date Thu, 09 Nov 2006 16:00:43 GMT
I looked at the source a bit and it looks like the buffer size might be
throttled.

I see that my base stream is an AutoCloseInputStream, which is created in
HttpMethodBase.
Its source is received from HttpConnection.getResposeInputStream().
It looks like the input stream is actually created in open() where I see
this code: (Version 3.01)

(The sndBufSize and rcvBufSize is controlled through HttpConnectionParams)

            int outbuffersize = socket.getSendBufferSize();
            if ((outbuffersize > 2048) || (outbuffersize <= 0)) {
                outbuffersize = 2048;
            }
            int inbuffersize = socket.getReceiveBufferSize();
            if ((inbuffersize > 2048) || (inbuffersize <= 0)) {
                inbuffersize = 2048;
            }
            inputStream = new BufferedInputStream(socket.getInputStream(),
inbuffersize);
            outputStream = new
BufferedOutputStream(socket.getOutputStream(), outbuffersize);

All this being said, even though the buffer I sent to read() is 64K I always
seem to read 8192 bytes at a time.  Looking at this code, I would expect to
be reading only 2048 bytes at a time.  I am still trying to figure this one
out.

Mark
 
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org] 
Sent: Thursday, November 09, 2006 6:11 AM
To: HttpClient User Discussion
Subject: Re: have question about buffer size

On Wed, 2006-11-08 at 22:27 +0200, Paranoid wrote:
> > Why should this be a problem? Over the network, maximum segments you 
> > get are (after removing framing/chunking overhead) probably that 
> > 1440 bytes in size, and httpclient is then returning you as much 
> > data as it can efficiently give at any given time.
> > You just keep on reading all the data, piece by piece.
> > That's how streams work; they present an abstraction over what may 
> > be (and often is) packet-based transport channel.
> 
> some peaces are about 55 KB size, and first read is 75 KB. so why are 
> all other about 1.4 KB?

There can be many factors affecting the content stream fragmentation and the
way it is being transferred across the wire. I suggest that you install a
traffic analyzer such as Ethereal and look at the packets sent by the target
server. If you are absolutely convinced the content gets fragmented
somewhere inside HttpClient, I'll dig into the code and try to pinpoint the
problem.

Oleg

> and after - saving inputStream into FileOutputStream load CPU for 
> great number... this time it is really a problem... tested with 
> different buffer sized, and with greater buffer size have lower CPU load.
> P.S.: speed of downloading about 5 MB/s...


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message