tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Crowther <Peter.Crowt...@melandra.com>
Subject RE: CLOSE_WAIT and what to do about it
Date Wed, 08 Apr 2009 11:40:23 GMT
> From: André Warnier [mailto:aw@ice-sa.com]
> It has been previously established that a socket in a
> long-time-lingering CLOSE-WAIT status, is due to one or the other side
> of a TCP connection not properly closing its side of the
> connection when
> it is done with it.
> I also surmise (without having a definite proof of this), that this is
> essentially "bad", as it ties up some resources that could be
> otherwise freed.

At the very least it'll tie up a kernel data structure for the socket itself.  I don't know
modern Linux kernels well enough to know how buffers are allocated, but I suspect you won't
be wasting much memory on buffers as they'll be allocated on-demand.  You're probably talking
tens to low hundreds of bytes for each one of these.

You will also be consuming resources in whichever program is not closing the sockets correctly.

> So my question is : considering the situation above, is there
> something
> I can do locally to free these lingering CLOSE_WAIT sockets, and under
> which conditions ?

> For example, I see this line :
> tcp6      12      0 ::ffff:127.0.0.1:41764  ::ffff:127.0.0.1:11002
> CLOSE_WAIT 29649/java
> which tells me that there is a local process 29649/java,
> whith a "local"
> socket port 41674 in the CLOSE_WAIT state, related to another socket
> #11002 on the same host.
> On the other hand, I see this line :
> tcp        0      0 127.0.0.1:11002         127.0.0.1:41764
> FIN_WAIT2  -
> which shows a "local" socket on port 11002, related to this
> other local
> socket port #41764, with no process-id/program displayed.
> What does that tell me ?

The process that was on port 11002 closed its end of the socket and sent a FIN.  Process 29649
hasn't closed its end of the socket yet.

> I also know that the process-id 29649 corresponds to a local java
> process, of the daemon variety, multi-threaded.  That program
> "talks to"
> another known server program, written in C, of which instances are
> started on an ad-hoc base by inetd, and which "listens" on port 11002
> (in fact it is inetd who does, and it passes this socket on to the
> process it forks, I understand that).

The local Java process may have a resource leak.  It appears not to have closed the socket
it was using to communicate with the server.

A possible reason for the lack of a PID on port 11002 is that the socket was handed across
from inetd to the C daemon - not sure about this.

> What it looks like to me in this case, is that at some point
> one of the
> threads of process # 29649 opened a client socket #41674 to the local
> inetd port #11002; that inetd then started the underlying
> server process
> (the C program); that the underlying C program then at some point
> exited; but that process #41674 never closes one of the sides of its
> connection with port #11002.

Agree.

> Can I somehow detect this condition, and "force" the
> offending thread of
> process #29649 to close that socket (or just force this
> thread to exit) ?

Threads are flows of control.  Threads do not reference objects other than from their stack
and any thread-local storage - and there are plenty of other places that can hold onto objects!
 The socket may well be referenced from an object on the heap (not the stack) that's ultimately
referenced by a static variable in a class, for example, in which case zapping a thread may
well do nothing.

You need to find out what, if anything, is holding onto the socket.

If you have some way of forcing that Java process to collect garbage, you should do so.  It's
possible for sockets that haven't been close()d to hang around, unreferenced but not yet garbage
collected.  A full GC would collect any of these, finalizing them as it does and hence closing
the socket.  If a full GC doesn't close the socket, some other object is still referencing
it.

If a full GC doesn't clear the problem, you may need to go in with some memory-tracing tool
and find out what's holding onto the socket.  It's a long, long time since I had to do this
in Java, so I have no idea of the appropriate tools - my brain's telling me Son of Strike,
which is for the .Net CLR and *definitely* wrong!

Does that help?  Or is it clear as mud?

                - Peter

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message