commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Sutton <>
Subject RE: [httpclient] MultiThreadedHttpConnectionManager : connections available
Date Sat, 12 Apr 2003 01:13:30 GMT
Hi Benoit,
It sounds like your making some not-so-recommended (but possible) use of
HttpClient.  There may well be a reason you need to do it that way that I
don't know about but if you can use the proceedure below I think you'd be
better off.

1. Create one instance of HttpClient with whatever connection manager you
want (the MultiThreadedConnectionManager sounds like it's the best).  Use
the setHttpConnectionFactoryTimeout method to set the timeout to something
very short (probably < 10ms).

2. Each thread is given a URL asynchronously and creates a HttpMethod
(probably GetMethod) for that URL, sets any configuration options that need
to be set, then passes the method into the execute method of the HttpClient
instance.  At most this will block for the timeout set in (1) and throw a
HttpException if a connection is not available - the thread can then dump
the URL back in the queue for later and move on to the next one.

The advantages of this approach are:

1. All connections are managed by HttpClient so you get the benefit of all
the user testing and fixes we've gone through in this area (and that's a
heck of a lot).

2. HttpClient handles all the synchronization and again you get the benefit
of other people maintaining that seciton of code.

3. HttpClient will handle a whole bunch of errors that routinely happen when
using Java's socket libraries.  It will also correctly handle chunked
encoding, cookies, authentication and a whole bunch of other stuff that
probably isn't relevant to your app but may be useful in the future. :)

Of course, if you didn't want any timeout at all, simply extend
MultiThreadedHttpConnectionManager so that it ignores the timeout value and
either returns immediately or throws an exception immediately, then pass
that in to the HttpClient constructor.

Having said that, the way you're doing it can definitely be made to work (I
know of at least 2 other people who ditched the HttpClient class itself and
managed connections and methods manually because they needed extra
configurability that HttpClient doesn't (yet) provide).  I just would avoid
it if possible since we've already gone to the trouble of getting it all
working so why not take full advantage of it? :)

Also, your general use case is almost exactly like the one that highlighted
the problems in MultiThreadedHttpConnectionManager that Mike is fixing so
you'll definitely want to upgrade to the nightly build once the patch is
committed.  I'd recommend creating a bugzilla account and adding yourself to
the CC list for bug 18596 so you get notifications when any changes to the
bug are made (like when Mike applies the now completed and working patch to
CVS). :)


Adrian Sutton.

-----Original Message-----
From: Benoit Mathieu []
Sent: Friday, 11 April 2003 5:25 PM
To: Jakarta Commons Users List
Subject: Re: [httpclient] MultiThreadedHttpConnectionManager :
connections available

Hi Adrian,

thanks for your explanations, i misunderstood the bug Mike is fixing.

Let's me give you some details about how I use HttpClient in my project:
I have a server which is provided with a list of URL to retrieve, this urls 
are given by other threads, asynchronously, and the server have to process 
Http requests and store results into files. I use a thread pool to process 
those requests. Of course, as httpClient does, the number of connection for
host is limited, so i'd like to know which connections are available in
not to have threads wasting time waiting for connections ... I don't know if

this kind of use honor the contracts of usage that HttpClient applies, but
me, it seems that httpClient would suit well my needs ;-)

The solution I choosed for the moment, is to extends 
MultiThreadedHttpConnectionManager by adding a non blocking version of 
getConnection() (which is very simple to write since it is the same as the 
blocking one, without timeOut and wait() ). When there is requests to
and a thread available, i try to get connections, and provide the thread
the first one i get. It seems to works until now ...

As you wrote, after a while my program is running out of memory. It may be 
because of the number of connections, so I'll try with the next version.

Thanks to the httpClient team, for their time, and their quality development



To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message