hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject RE: have question about buffer size
Date Thu, 09 Nov 2006 16:21:59 GMT
On Thu, 2006-11-09 at 11:00 -0500, Mark Claassen wrote:
> I looked at the source a bit and it looks like the buffer size might be
> throttled.
> 
> I see that my base stream is an AutoCloseInputStream, which is created in
> HttpMethodBase.
> Its source is received from HttpConnection.getResposeInputStream().
> It looks like the input stream is actually created in open() where I see
> this code: (Version 3.01)
> 
> (The sndBufSize and rcvBufSize is controlled through HttpConnectionParams)
> 
>             int outbuffersize = socket.getSendBufferSize();
>             if ((outbuffersize > 2048) || (outbuffersize <= 0)) {
>                 outbuffersize = 2048;
>             }
>             int inbuffersize = socket.getReceiveBufferSize();
>             if ((inbuffersize > 2048) || (inbuffersize <= 0)) {
>                 inbuffersize = 2048;
>             }
>             inputStream = new BufferedInputStream(socket.getInputStream(),
> inbuffersize);
>             outputStream = new
> BufferedOutputStream(socket.getOutputStream(), outbuffersize);
> 
> All this being said, even though the buffer I sent to read() is 64K I always
> seem to read 8192 bytes at a time.  Looking at this code, I would expect to
> be reading only 2048 bytes at a time.  I am still trying to figure this one
> out.
> 


Mark,

The way BufferedInputStream#read(byte[], int, int) is implemented
content looks quite intelligent to me. If there is not content in the
intermediate buffer BufferedInputStream will read directly from the
underlying input stream. So, the size of the intermediate buffer should
not matter a lot, when not reading content one byte at a time 

Oleg


> Mark
>  
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org] 
> Sent: Thursday, November 09, 2006 6:11 AM
> To: HttpClient User Discussion
> Subject: Re: have question about buffer size
> 
> On Wed, 2006-11-08 at 22:27 +0200, Paranoid wrote:
> > > Why should this be a problem? Over the network, maximum segments you 
> > > get are (after removing framing/chunking overhead) probably that 
> > > 1440 bytes in size, and httpclient is then returning you as much 
> > > data as it can efficiently give at any given time.
> > > You just keep on reading all the data, piece by piece.
> > > That's how streams work; they present an abstraction over what may 
> > > be (and often is) packet-based transport channel.
> > 
> > some peaces are about 55 KB size, and first read is 75 KB. so why are 
> > all other about 1.4 KB?
> 
> There can be many factors affecting the content stream fragmentation and the
> way it is being transferred across the wire. I suggest that you install a
> traffic analyzer such as Ethereal and look at the packets sent by the target
> server. If you are absolutely convinced the content gets fragmented
> somewhere inside HttpClient, I'll dig into the code and try to pinpoint the
> problem.
> 
> Oleg
> 
> > and after - saving inputStream into FileOutputStream load CPU for 
> > great number... this time it is really a problem... tested with 
> > different buffer sized, and with greater buffer size have lower CPU load.
> > P.S.: speed of downloading about 5 MB/s...
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


Mime
View raw message