hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: CLOSE_WAIT on Linux
Date Fri, 31 Jul 2009 11:21:26 GMT
On Thu, Jul 30, 2009 at 10:47:06AM -0700, Gerry Woods wrote:
> Oleg,
> Thanks very much for the reply.  I mentioned the SO_TIMEOUT setting just to explain the
condition that led to the exception and seems to have ended up leaving the socket in a bad
state.  I understand too that CLOSE_WAIT is due to the server having closed the socket but
httpclient has not.  We do run the idle connection timeout thread, and unfortunately in this
case it doesn't seem to help.  
> 
> The focus of my attention has been the HttpConnection.closeSocketAndStreams() method
which is where the socket seems to end up being disposed of both by the IdleConnectionTimeoutThread
and all other logic that closes the connection.  In 3.0.1 and 3.1, this simply calls close()
on the streams and sockets.  In 4.x however, the SocketHttpClientConnection.close() method
does perform a shutdownOutput()/shutdownInput().  Some snippets of information I've found
seem to indicate that on some socket stacks close() is not enough to actually close a socket
and make the CLOSE_WAIT->LAST_ACK transition, and that a shutdown is necessary.  This seems
to jive with our experience here. I was assuming that the change in behavior from 3.x to 4.x
was the result of lessons learned and that maybe this had a bearing on our scenario.  If not,
we can run some experiments and try to isolate it further.  Eventually of course we would
like to migrate to the newer, cleaner, APIs.  But in the meantime if anything occurs to you
or anyone else about this condition, please let me know.
> Thanks again.
> Gerry
> 

Gerry,

The Socket#shutdown*() methods have become available since Java 1.3 only.
HttpClient 3.x cannot make use of those methods due to java 1.2.2 compatibility
requirement.

http://java.sun.com/j2se/1.4.2/docs/api/java/net/Socket.html#shutdownInput%28%29
http://java.sun.com/j2se/1.4.2/docs/api/java/net/Socket.html#shutdownOutput%28%29

I am still not convinced Socket#close() does not fully close the underlying TCP
connection. I think shutdown methods are needed only if one wants to put a
socket to a half-closed state (read-only / write-only).

Oleg



> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org] 
> Sent: Thursday, July 30, 2009 3:08 AM
> To: HttpClient User Discussion
> Subject: Re: CLOSE_WAIT on Linux
> 
> On Wed, Jul 29, 2009 at 09:36:20PM -0700, Gerry Woods wrote:
> > I haven't posted here before so let me start by thanking you guys for the great
work.
> > 
> > We are using 3.0.1 in some pretty heavy production environments.   A recent issue
arose with a customer who had trouble during SSL handshakes.  The negotiation was timing out:
> > 
> >  [120009] java.net.SocketTimeoutException: Read timed out
> >                 at java.net.SocketInputStream.socketRead0(Native Method)
> >                 at java.net.SocketInputStream.read(Unknown Source)
> >                 at com.sun.net.ssl.internal.ssl.InputRecord.a(Unknown Source)
> >                 at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> >                 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(Unknown Source)
> >                 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.j(Unknown Source)
> >                 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.b(Unknown Source)
> >                 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown
Source)
> >                 ...
> > 
> > We're setting SO_TIMEOUT to 2 minutes.  The customer reports that the machine that
initiated the connection ended up with a lot of sockets in CLOSE_WAIT.  From what I've been
able to glean from some quick searches, it sounds like on some socket stacks, a shutdownOutput
is required, and that close() is not enough to nudge the socket from CLOSE_WAIT to LAST_ACK.
 I notice that in the 4.x codebase, shutdownOutput is being used, but in the 3.x code it is
not.  Can anyone confirm that this is indeed the case for Linux?  If so, is there a known
workaround other than upgrading to 4.x?
> > 
> > Thanks for any help you can offer, and thanks again for all the countless hours
(and dollars!) your efforts have saved us.
> > 
> >
> 
> Gerry,
> 
> As far as I know SO_TIMEOUT parameter and CLOSE_WAIT socket state are
> completely unrelated. Usually sockets end up in the CLOSE_WAIT state when a
> persistent connection is closed by the remote server while still kept alive in
> the connection pool on the client. This is general problem with the blocking
> I/O model and both HttpClient 3.x and 4.0 are affected by it. However,
> HttpClient 4.0 is much better equipped to deal with the issue. In both cases
> one needs to implement some sort of connection eviction policy to make sure
> that persistent connections that have been idle for too long get closed and
> evicted from the pool.
> 
> http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e638
> 
> Hope this helps
> 
> Oleg
> 
>  
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message