tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: CLOSE_WAIT and what to do about it
Date Sun, 12 Apr 2009 13:21:32 GMT
Caldarale, Charles R wrote:
>> From: André Warnier [mailto:aw@ice-sa.com]
>> Subject: Re: CLOSE_WAIT and what to do about it
>>
>> Relatedly, does there exist any way to force a given 
>> JVM process to do a full GC interactively, but from a
>> Linux command-line ?
> 
> Found a command line tool that will do what you want:
> http://code.google.com/p/jmxsh/
> 
> I've used it to trigger a GC in Tomcat via the following steps.
> 
> 1) Start Tomcat with the following options:
>      -Dcom.sun.management.jmxremote.port=<port>
>      -Dcom.sun.management.jmxremote.authenticate=false
>      -Dcom.sun.management.jmxremote.ssl=false
>    (You can, of course, set the authentication and SSL options as needed.)
> 
> 2) Start jmxsh from the directory its jar is in with this:
>       java -jar jmxsh*.jar
> 
> 3) Enter the following commands (but not the bracketed bits):
>       jmx_connect -h localhost -p <port>
>       [blank line to enter browse mode]
>       5  [selects java.lang]
>       1  [selects the Memory mbean]
>       5  [performs a GC]
> 
> The doc for jmxsh indicates the above steps should be scriptable, but I haven't tried
that.
> 
> It is likely that you could use jmx_connect with a different kind of service and avoid
opening up an RMI port; if I figure that out, I'll let you know.
> 

Hi.
Thanks a million for providing the above info.
That jmxsh program is really useful.
I don't really know what I'm doing here, but I can at least more or less 
figure out what happens.

To recall, my original issue is that I have some Java applications 
(among which a Tomcat webapp and a couple of stand-alone Java 
daemon-like programs) which apparently leave an ever-increasing number 
of sockets lingering in a CLOSE_WAIT state.
And I was wondering if it was possible, as one test, to force the JVM 
running these applications to perform a GC, right now, from "the outside".
Well, it is.

Following is a trace of a session with jmxsh, with one of these 
applications.


Initial socket situation :

root@arthur:/home/star/xml# netstat -pan | grep CLOSE
tcp6       0      0 ::ffff:127.0.0.1:48267  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7618/java
tcp6      12      0 ::ffff:127.0.0.1:36936  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7816/java
tcp6      12      0 ::ffff:127.0.0.1:50322  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7816/java

root@arthur:/home/star/xml# ps -ef | grep 7618
root      7618     1  1 14:32 pts/3    00:00:15 ./java -server 
-Dcom.sun.management.jmxremote.port=11201 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false -Xms64M -Xmx64M 
-Dpgm=STARWeb -jar /home/xxxx/web4/java/xyz.jar -c 
/home/star/web4/config -p 11101

The above is the process which I am going to "stress", in the sense of 
communicating with it, which has the result of having it itself open a 
TCP connection with another server listening on port 11002, then closing 
this socket (in principle), and this multiple times.

(As you see, the program was started with the "jmxremote" options 
allowing later communication with jmxsh.)

Now some interactions with the application pid=7618 ...
Situation later on :

root@arthur:/home/star/xml# netstat -pan | grep CLOSE
tcp6       0      0 ::ffff:127.0.0.1:55798  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7618/java
tcp6       0      0 ::ffff:127.0.0.1:57029  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7618/java
tcp6       0      0 ::ffff:127.0.0.1:48267  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7618/java
tcp6       0      0 ::ffff:127.0.0.1:56781  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7618/java
tcp6      12      0 ::ffff:127.0.0.1:36936  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7816/java
tcp6      12      0 ::ffff:127.0.0.1:58341  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7816/java
tcp6       0      0 ::ffff:127.0.0.1:32972  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7618/java
tcp6      12      0 ::ffff:127.0.0.1:50322  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7816/java

So this application indeed left a number of sockets in the CLOSE_WAIT state.

Now triggering a GC with jmxsh :

aw@arthur:~$ java -jar bin/jmxsh-R4.jar
jmxsh v1.0, Tue Jan 22 17:23:12 GMT+01:00 2008

Type 'help' for help.  Give the option '-?' to any command
for usage help.

Starting up in shell mode.
% jmx_connect -h localhost -p 11201
Connected to service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi.

%
Entering browse mode.
====================================================

  Available Domains:

        1. java.util.logging
        2. JMImplementation
        3. java.lang

   SERVER: service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi

====================================================
Select a domain: 3
====================================================

  Available MBeans:

        1. java.lang:type=Compilation
        2. java.lang:type=MemoryManager,name=CodeCacheManager
        3. java.lang:type=GarbageCollector,name=Copy
        4. java.lang:type=MemoryPool,name=Eden Space
        5. java.lang:type=Runtime
        6. java.lang:type=ClassLoading
        7. java.lang:type=MemoryPool,name=Survivor Space
        8. java.lang:type=Threading
        9. java.lang:type=GarbageCollector,name=MarkSweepCompact
       10. java.lang:type=OperatingSystem
       11. java.lang:type=Memory
       12. java.lang:type=MemoryPool,name=Code Cache
       13. java.lang:type=MemoryPool,name=Tenured Gen
       14. java.lang:type=MemoryPool,name=Perm Gen

   SERVER: service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi
   DOMAIN: java.lang

====================================================
Select an mbean: 11
====================================================

  Attribute List:

        1. -r- javax.management.openmbean.CompositeData 
NonHeapMemoryUsage
        2. irw boolean     Verbose
        3. -r- int         ObjectPendingFinalizationCount
        4. -r- javax.management.openmbean.CompositeData 
HeapMemoryUsage

  Operation List:

        5. void        gc()

   SERVER: service:jmx:rmi:///jndi/rmi://localhost:11201/jmxrmi
   DOMAIN: java.lang
   MBEAN:  java.lang:type=Memory

====================================================
Select an attribute or operation:  5
=====================================================

    Invoking Operation gc

Invoked.
Press enter to continue.
====================================================
Select an attribute or operation:
Entering shell mode.
% jmx_close

% exit
aw@arthur:~$

Socket situation after the GC :

root@arthur:/home/star/xml# netstat -pan | grep CLOSE
tcp6       0      0 ::ffff:127.0.0.1:55798  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7618/java
tcp6      12      0 ::ffff:127.0.0.1:36936  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7816/java
tcp6      12      0 ::ffff:127.0.0.1:58341  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7816/java
tcp6      12      0 ::ffff:127.0.0.1:50322  ::ffff:127.0.0.1:11002 
CLOSE_WAIT 7816/java
root@arthur:/home/star/xml#

And indeed all but one of the CLOSE_WAIT sockets of process-id 7618 have 
disappeared.

=========================================

I will now risk an interpretation :
If these sockets disappear during a GC, then it must mean that they are 
still being referenced by some "abandoned" objects sitting on the Heap, 
which have not yet been reclaimed by the GC.
Which probably means that the objects in question have gone out of 
scope, before the socket they used was properly close()'d.

Yes ?



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message