hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: performance tuning possibilities?
Date Fri, 10 Jul 2009 09:14:40 GMT
On Thu, Jul 09, 2009 at 08:27:33PM +0200, Michael Gnatz wrote:
> OK, Oleg's suggestions improved the whole thing around 30%, great stuff!
> 
> Where would the performance boost with using the Core directly come from?
> 

HttpCore implements just a very basic HTTP transport. It can transfer HTTP
messages across the wire very efficiently, but it cannot (and is not supposed
to) do anything beyond that. HttpClient adds connection management, state
management, authentication and rediect handling logic on top of HttpCore. All
that extra functionality results in a ~30% performance penalty. If you do not
mind managing connections manually, do not need cookie support, authentication
and automatic redirects, you may well consider using HttpCore directly.

Oleg


> 
> 2009/7/9 Oleg Kalnichevski <olegk@apache.org>
> 
> > On Thu, Jul 09, 2009 at 05:05:14PM +0200, Michael wrote:
> > > I completed my migration to the latest version of the HttpClient
> > > successfully. It is just a bit slower now than I would have ecpected...
> > >
> > > Maybe somebody has some tuning experiences to share?
> > >
> >
> > Michael,
> >
> > HttpClient should be _at least_ as fast as HttpClient 3.x. If performance
> > is an
> > overriding factor for you, you should consider using HttpCore directly and
> > get
> > 30 to 40% performance boost.
> >
> > http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore
> >
> >
> >
> > > * Reliability is not that important for me
> >
> > Make sure to turn off stale connection check and expect-continue handshake.
> >
> > > * It should do around 20 accesses per minute using slow proxies,
> > connection
> > > timeout is set to 6 seconds for them. Currently it does those 20 accesses
> > in
> > > 100 seconds :-(
> > >
> >
> > To find out where time is being spent, turn on context logging. Make sure
> > to
> > disable the wire log, though, otherwise you will be measuring performance
> > of
> > console output.
> >
> > http://hc.apache.org/httpcomponents-client/logging.html
> >
> >
> > >
> > > * I have 3 threads executing http gets concurrently.
> > > * I am going with ThreadSafeClientConnManager and 30 connections at max.
> > Is
> > > that enough? Or too much?
> >
> > You also have to increase the default limit of 2 concurrent connections per
> > host. It should be at least the same as the number of worker thread to
> > ensure
> > minimal resource contension.
> >
> >
> > > * I use a retry hanlder with up to 6 retries. Not sure if that is eating
> > up
> > > much time?
> > > * TCPNoDelay is false. Would turning that on improve performance to some
> > > extent? How much?
> >
> > Whooooaaaaa! Nagle's algorithm just kills HTTP performance. Make sure the
> > TCPNoDelay is set to true.
> >
> > > * Stale Checking is enabled. Maybe I dont need this since I use the
> > > IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every
> > 5
> > > seconds?
> > >
> >
> > The stale connection check is more evil than the Church of Scientology.
> > Just do
> > not use it.
> >
> > Hope this helps
> >
> > Oleg
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message