Mailing-List: contact commons-httpclient-dev-help@jakarta.apache.org;
 run by ezmlm
Precedence: bulk
Reply-To: "Commons HttpClient Project"
 <commons-httpclient-dev@jakarta.apache.org>
Received-SPF: pass (hermes.apache.org: local policy)
Subject: Re: threads problem with many connections
From: Oleg Kalnichevski <oleg.kalnichevski@bearingpoint.com>
To: Commons HttpClient Project <commons-httpclient-dev@jakarta.apache.org>
In-Reply-To: <87r7oi3dud.fsf@meuh.mnc.ch>
References: <87r7oi3dud.fsf@meuh.mnc.ch>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Message-Id: <1096630045.2486.20.camel@localhost.localdomain>
Mime-Version: 1.0
Date: Fri, 01 Oct 2004 13:27:25 +0200


On Fri, 2004-10-01 at 12:57, Guillaume Cottenceau wrote:
> Hello,
> 
> We use HttpClient for performing several HTTP post in parallel in
> our applications. We have a problem when the server(s) receiving
> our HTTP post either answers very slowly, or goes mad and sends
> garbage data over and over: the connection stays open forever,
> but more important, the Java threads as well. We have situations
> where we reach the maximum of Java threads our (tomcat)
> application is configured to handle, and our whole application is
> then unusable.
> 
> It seems that java.nio is capable of using only one thread for
> several lowlevel (OS) socket connections, and is actually also
> quite efficient.

Guillaume, 

Please correct me if my understanding of the problem is incorrect, it is
Tomcat that runs out of connections, not HttpClient. In fact, there's
only one case when HttpClient spawns a new thread, and if so is desired,
this can be disabled. The thread management is entirely responsibility
of your application. One does not have to use one thread per socket
model even without NIO 

> 
> I have seen that Oleg Kalnichevski has already expressed his
> views several times on the subject, and I have seen that you want
> to keep 1.2 compatibility, so java.nio out.
> 
> http://www.mail-archive.com/commons-httpclient-dev@jakarta.apache.org/msg05551.html
> http://www.mail-archive.com/commons-httpclient-dev@jakarta.apache.org/msg06998.html
> 
> (btw, I don't agree with "in my opinion there's absolutely
> nothing that NIO can bring in in terms of performance to
> client-side applications" - well I agree that pure "performance"
> is not the problem but threads and memory consumption surely is -
> so in my opinion there is a lot to win with java.nio in
> httpclient)
> 

I have done quite a bit of NIO programming lately and respectfully stick
to my original opinion.

> My question is, since you don't want to lose 1.2 compatibility
> before 4.0, is there then a way to solve a typical "too many
> threads" problem such as the one we have? Do you people never had
> the same problem? Or have found a way to solve it?

Feasible approach is to have one monitor thread checking on the status
of active connections or/and processing incoming connections, and a
number of worker threads in a shared pool to do the actual work.

> 
> It seems the HTTP protocol doesn't have anything resembling a
> "global timeout" for a given connection (e.g. after x seconds,
> close the receiving channel even if server hasn't finished
> sending), and thus normally httpclient doesn't provide such a
> thing. Do you think this should be investigated/implemented in
> some way?

I do not recall HTTP protocol as such dealing with timeouts in any way.
I may be wrong, though. This is certainly not a protocol concern.
HttpClient does provide so called socket timeout. When set to a positive
number, the connection will terminate with an IOException after the
specified period of inactivity:

http://jakarta.apache.org/commons/httpclient/apidocs/org/apache/commons/httpclient/HttpClient.html#setTimeout(int) 
Hope this helps

Oleg

> 
> Thanks, and best regards.

***************************************************************************************************
The information in this email is confidential and may be legally privileged.  Access to this email by anyone other than the intended addressee is unauthorized.  If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful.  If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.
***************************************************************************************************

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org