tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: Issue with Connections
Date Wed, 06 Feb 2013 20:11:37 GMT
TVFoodMaps wrote:
> Thanks for the information.  I'm still a bit perplexed how to
> increase/change the settings so these issues dont cause the "hang" I saw
> this morning.  My guess was that I hit some type of limit on sockets/file
> descriptors.  I'm not sure if the fix is in my application, network
> settings or the OS file descriptor limits.
> 
> On a related note, I also notice lots of TIME_WAIT connections to port
> 8080.  What controls how long the connection is in TIME_WAIT?

Here is a very good article about it, with pictures and all.
http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html


> 
> 
> On Wed, Feb 6, 2013 at 12:32 PM, André Warnier <aw@ice-sa.com> wrote:
> 
>> André Warnier wrote:
>>
>>> TVFoodMaps wrote:
>>>
>>>> Hi,
>>>>
>>>> My website is setup using apache 2, mod jk 1.37 and tomcat 6.  Most of
>>>> the
>>>> connector settings are set to defaults and things normally run pretty
>>>> well.
>>>>  However during certain spikes in traffic my server seems to hang by what
>>>> appears to be caused by "leaking connections".  What I see is about 250
>>>> rows like this from a netstat:
>>>>
>>>> tcp        1      0 ::ffff:127.0.0.1:8009       ::ffff:127.0.0.1:48387
>>>>  CLOSE_WAIT
>>>> tcp        1      0 ::ffff:127.0.0.1:8009       ::ffff:127.0.0.1:48309
>>>>  CLOSE_WAIT
>>>> tcp      689      0 ::ffff:127.0.0.1:8009       ::ffff:127.0.0.1:48423
>>>>  CLOSE_WAIT
>>>> tcp      686      0 ::ffff:127.0.0.1:8009       ::ffff:127.0.0.1:48413
>>>>  CLOSE_WAIT
>>>>
>>>> I also am see a lot of these:
>>>>
>>>> tcp        1      0 ::ffff:127.0.0.1:49261      ::ffff:127.0.0.1:8080
>>>> CLOSE_WAIT
>>>> tcp        1      0 ::ffff:127.0.0.1:52836      ::ffff:127.0.0.1:8080
>>>> CLOSE_WAIT
>>>> tcp        0      0 ::ffff:127.0.0.1:58262      ::ffff:127.0.0.1:8080
>>>> TIME_WAIT
>>>>
>>>> (Note the application makes direct calls to port 8080 for a specific API
>>>> I'm using (SOLR)).
>>>>
>>>> I'm really not sure which is actually causing the problem, but I was
>>>> hoping
>>>> for some guidane on which settings I should look into tweaking.
>>>>
>>>>
>>>>  Hi.
>>> 1) here is one (among many) explanation of the CLOSE_WAIT state.
>>> http://blogs.technet.com/b/**janelewis/archive/2010/03/09/**
>>> explaining-close-wait.aspx<http://blogs.technet.com/b/janelewis/archive/2010/03/09/explaining-close-wait.aspx>
>>> (and many more if you search google for "tcp close_wait")
>>> Basically, it is a normal state through which any TCP connection passes
>>> at some point.
>>> It is only pathological when you many of them persisting for a long time.
>>>
>>> 2) assuming yours are pathological and persist a long time :
>>> 2.a) the ones like this :
>>>  > tcp        1      0 ::ffff:127.0.0.1:8009       ::ffff:127.0.0.1:48387
>>>  >  CLOSE_WAIT
>>>
>>> involve port 8009, which is the AJP port of Tomcat, in this case the
>>> "server" side. The other side is mod_jk within Apache httpd, in this case
>>> the client side (because it is mod_jk which first establishes the
>>> connection to Tomcat, so mod_jk is the client here).
>>> If one of these persists for a long time, it means that the client
>>> (mod_jk) does not entirely close its connection to Tomcat.
>>> Why that could be, another person here would have to explain.
>>>
>>> 2.b) the other ones like
>>>  > tcp        1      0 ::ffff:127.0.0.1:49261      ::ffff:127.0.0.1:8080
>>>  > CLOSE_WAIT
>>>
>>> relate to your application (as a client), which does not entirely close()
>>> its connections to port 8080.
>>> In my own experience - and assuming that you application is a java
>>> application - this can happen for example as follows :
>>> - the explicit connection to port 8080 is made from within some object,
>>> as part of the creation of that object
>>> - then when the application doesn't need the "connection object" anymore,
>>> it discards it.
>>> - the object is left on the heap, waiting to be garbage-collected
>>> - when it is (eventually) garbage-collected, it will really de destroyed,
>>> and any lingering "socket" within it will be closed (and the line above
>>> will disappear from your netstat output)
>>> but..
>>> while it sits on the heap waiting to be garbage-collected (which could be
>>> for a long time if your system has a lot of spare heap memory), that inner
>>> socket is still there, not totally closed (the server closed its side, but
>>> the client didn't).
>>>
>>> Eventually, you may have so many sockets in the CLOSE_WAIT state, that
>>> your system's TCP stack becomes unresponsive. (That's what I have seen
>>> happening under Linux).
>>>
>>> I do not really know the real underlying reason for this behaviour, but
>>> my guess is that below the level of Java and the JVM, a Java socket at some
>>> level relies on a OS-level socket object.  And as long as that OS-level
>>> socket object is not explicitly told to close() the connection, it doesn't.
>>> So make sure that before you discard your high-level objects containing a
>>> connection, you explicitly close that connection.
>>>
>>>
>>>  Addendum : this may be interesting too :
>> http://www.michaelprivat.com/?**p=63 <http://www.michaelprivat.com/?p=63>
>>
>>
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.**apache.org<users-unsubscribe@tomcat.apache.org>
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message