hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Claassen <mclaas...@ocie.net>
Subject RE: have question about buffer size
Date Thu, 09 Nov 2006 19:34:13 GMT
I am glad to hear that 4.0 is going to be even faster, but I was more
interested in figuring out what was going on.  I am actually very pleased
with the performance of HttpClient over the java.net stuff.  I have not
tested raw stream performance, but in all other ways it is certainly far
superior.

Mark
 
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org] 
Sent: Thursday, November 09, 2006 12:23 PM
To: HttpClient User Discussion
Subject: RE: have question about buffer size

On Thu, 2006-11-09 at 11:41 -0500, Mark Claassen wrote:
> I just finished looking at that a bit, and I see that now that you do 
> read a byte at a time.  Knowing this, I agree with you.
> 
> I am now focusing on chunked input stream.  I don't really know how 
> these things work, but it seems that the chunk size is sent every time 
> (?) and you read the size and then read the data.  Looking at my chunk 
> sizes, is see that they are 8K or less.  This corresponds to the behavior
I am seeing.
> 
> I must confess that I have no idea if any optimization would be 
> possible here.  I guess you could read more than one chunk at a time, 
> but I don't know if that would help anything.
> 

Mark,

It is not worthwhile investing any efforts into optimizing it because
HttpClient 3.x code line will no longer be actively developed past 3.1
release. HttpClient 4.0 will be based on HttpCore [1][2] which has much more
memory efficient and performant buffering code among other things.
Overall I expect HttpClient 4.0 be 40 to 50% faster and by an order of
magnitude more memory efficient than HttpClient 3.1 due to its low lever
transport code (HttpCore)

[1] http://jakarta.apache.org/httpcomponents/index.html
[2] http://jakarta.apache.org/httpcomponents/http-core/index.html

Oleg

> Mark
>  
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Thursday, November 09, 2006 11:22 AM
> To: HttpClient User Discussion
> Subject: RE: have question about buffer size
> 
> On Thu, 2006-11-09 at 11:00 -0500, Mark Claassen wrote:
> > I looked at the source a bit and it looks like the buffer size might 
> > be throttled.
> > 
> > I see that my base stream is an AutoCloseInputStream, which is 
> > created in HttpMethodBase.
> > Its source is received from HttpConnection.getResposeInputStream().
> > It looks like the input stream is actually created in open() where I 
> > see this code: (Version 3.01)
> > 
> > (The sndBufSize and rcvBufSize is controlled through
> > HttpConnectionParams)
> > 
> >             int outbuffersize = socket.getSendBufferSize();
> >             if ((outbuffersize > 2048) || (outbuffersize <= 0)) {
> >                 outbuffersize = 2048;
> >             }
> >             int inbuffersize = socket.getReceiveBufferSize();
> >             if ((inbuffersize > 2048) || (inbuffersize <= 0)) {
> >                 inbuffersize = 2048;
> >             }
> >             inputStream = new
> > BufferedInputStream(socket.getInputStream(),
> > inbuffersize);
> >             outputStream = new
> > BufferedOutputStream(socket.getOutputStream(), outbuffersize);
> > 
> > All this being said, even though the buffer I sent to read() is 64K 
> > I always seem to read 8192 bytes at a time.  Looking at this code, I 
> > would expect to be reading only 2048 bytes at a time.  I am still 
> > trying to figure this one out.
> > 
> 
> 
> Mark,
> 
> The way BufferedInputStream#read(byte[], int, int) is implemented 
> content looks quite intelligent to me. If there is not content in the 
> intermediate buffer BufferedInputStream will read directly from the 
> underlying input stream. So, the size of the intermediate buffer 
> should not matter a lot, when not reading content one byte at a time
> 
> Oleg
> 
> 
> > Mark
> >  
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Thursday, November 09, 2006 6:11 AM
> > To: HttpClient User Discussion
> > Subject: Re: have question about buffer size
> > 
> > On Wed, 2006-11-08 at 22:27 +0200, Paranoid wrote:
> > > > Why should this be a problem? Over the network, maximum segments 
> > > > you get are (after removing framing/chunking overhead) probably 
> > > > that 1440 bytes in size, and httpclient is then returning you as 
> > > > much data as it can efficiently give at any given time.
> > > > You just keep on reading all the data, piece by piece.
> > > > That's how streams work; they present an abstraction over what 
> > > > may be (and often is) packet-based transport channel.
> > > 
> > > some peaces are about 55 KB size, and first read is 75 KB. so why 
> > > are all other about 1.4 KB?
> > 
> > There can be many factors affecting the content stream fragmentation 
> > and the way it is being transferred across the wire. I suggest that 
> > you install a traffic analyzer such as Ethereal and look at the 
> > packets sent by the target server. If you are absolutely convinced 
> > the content gets fragmented somewhere inside HttpClient, I'll dig 
> > into the code and try to pinpoint the problem.
> > 
> > Oleg
> > 
> > > and after - saving inputStream into FileOutputStream load CPU for 
> > > great number... this time it is really a problem... tested with 
> > > different buffer sized, and with greater buffer size have lower 
> > > CPU
> load.
> > > P.S.: speed of downloading about 5 MB/s...
> > 
> > 
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: 
> > httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: 
> > httpclient-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> httpclient-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> httpclient-user-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


Mime
View raw message