hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerry Woods <Gerry.Wo...@soa.com>
Subject RE: CLOSE_WAIT on Linux
Date Thu, 30 Jul 2009 17:47:06 GMT
Oleg,
Thanks very much for the reply.  I mentioned the SO_TIMEOUT setting just to explain the condition
that led to the exception and seems to have ended up leaving the socket in a bad state.  I
understand too that CLOSE_WAIT is due to the server having closed the socket but httpclient
has not.  We do run the idle connection timeout thread, and unfortunately in this case it
doesn't seem to help.  

The focus of my attention has been the HttpConnection.closeSocketAndStreams() method which
is where the socket seems to end up being disposed of both by the IdleConnectionTimeoutThread
and all other logic that closes the connection.  In 3.0.1 and 3.1, this simply calls close()
on the streams and sockets.  In 4.x however, the SocketHttpClientConnection.close() method
does perform a shutdownOutput()/shutdownInput().  Some snippets of information I've found
seem to indicate that on some socket stacks close() is not enough to actually close a socket
and make the CLOSE_WAIT->LAST_ACK transition, and that a shutdown is necessary.  This seems
to jive with our experience here. I was assuming that the change in behavior from 3.x to 4.x
was the result of lessons learned and that maybe this had a bearing on our scenario.  If not,
we can run some experiments and try to isolate it further.  Eventually of course we would
like to migrate to the newer, cleaner, APIs.  But in the meantime if anything occurs to you
or anyone else about this condition, please let me know.
Thanks again.
Gerry

-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org] 
Sent: Thursday, July 30, 2009 3:08 AM
To: HttpClient User Discussion
Subject: Re: CLOSE_WAIT on Linux

On Wed, Jul 29, 2009 at 09:36:20PM -0700, Gerry Woods wrote:
> I haven't posted here before so let me start by thanking you guys for the great work.
> 
> We are using 3.0.1 in some pretty heavy production environments.   A recent issue arose
with a customer who had trouble during SSL handshakes.  The negotiation was timing out:
> 
>  [120009] java.net.SocketTimeoutException: Read timed out
>                 at java.net.SocketInputStream.socketRead0(Native Method)
>                 at java.net.SocketInputStream.read(Unknown Source)
>                 at com.sun.net.ssl.internal.ssl.InputRecord.a(Unknown Source)
>                 at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
>                 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(Unknown Source)
>                 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.j(Unknown Source)
>                 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.b(Unknown Source)
>                 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown
Source)
>                 ...
> 
> We're setting SO_TIMEOUT to 2 minutes.  The customer reports that the machine that initiated
the connection ended up with a lot of sockets in CLOSE_WAIT.  From what I've been able to
glean from some quick searches, it sounds like on some socket stacks, a shutdownOutput is
required, and that close() is not enough to nudge the socket from CLOSE_WAIT to LAST_ACK.
 I notice that in the 4.x codebase, shutdownOutput is being used, but in the 3.x code it is
not.  Can anyone confirm that this is indeed the case for Linux?  If so, is there a known
workaround other than upgrading to 4.x?
> 
> Thanks for any help you can offer, and thanks again for all the countless hours (and
dollars!) your efforts have saved us.
> 
>

Gerry,

As far as I know SO_TIMEOUT parameter and CLOSE_WAIT socket state are
completely unrelated. Usually sockets end up in the CLOSE_WAIT state when a
persistent connection is closed by the remote server while still kept alive in
the connection pool on the client. This is general problem with the blocking
I/O model and both HttpClient 3.x and 4.0 are affected by it. However,
HttpClient 4.0 is much better equipped to deal with the issue. In both cases
one needs to implement some sort of connection eviction policy to make sure
that persistent connections that have been idle for too long get closed and
evicted from the pool.

http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e638

Hope this helps

Oleg

 

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message