tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Date Thu, 14 Mar 2013 12:11:19 GMT
David Kumar wrote:
> Hey,
> 
> I'm using: 
> 
> lsof -u tomcat
> 

example of "netstat" output on one of our own systems right now :

1) first one

vovm1:~# netstat -t -pan | grep -v ESTAB
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State 
PID/Program name
tcp        0      0 0.0.0.0:32032           0.0.0.0:*               LISTEN      2999/usbsrvd
tcp        0      0 0.0.0.0:20224           0.0.0.0:*               LISTEN      2965/inetd
tcp        0      0 0.0.0.0:11022           0.0.0.0:*               LISTEN      9179/perl
tcp        0      0 127.0.0.1:111           0.0.0.0:*               LISTEN      2172/portmap
tcp        0      0 0.0.0.0:54321           0.0.0.0:*               LISTEN      18279/python
tcp        0      0 0.0.0.0:4949            0.0.0.0:*               LISTEN 
23225/munin-node
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2431/sshd
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN      2965/inetd
tcp        0      0 0.0.0.0:43961           0.0.0.0:*               LISTEN      -
tcp        0      0 192.168.20.43:25        0.0.0.0:*               LISTEN      2946/exim4
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2946/exim4
tcp        0      0 0.0.0.0:11002           0.0.0.0:*               LISTEN      2965/inetd
tcp        0      0 0.0.0.0:46911           0.0.0.0:*               LISTEN      2183/rpc.statd
tcp        0      0 127.0.0.1:11002         127.0.0.1:40739         TIME_WAIT   -
tcp        0      0 127.0.0.1:11002         127.0.0.1:40735         TIME_WAIT   -
tcp        0      0 127.0.0.1:11002         127.0.0.1:40728         TIME_WAIT   -
tcp        0      0 127.0.0.1:11002         127.0.0.1:40724         TIME_WAIT   -
tcp        1      0 127.0.0.1:54321         127.0.0.1:37916         CLOSE_WAIT  18279/python
tcp        0      0 127.0.0.1:43797         127.0.0.1:8009          TIME_WAIT   -
tcp        0      0 192.168.20.43:56603     192.168.20.80:111       TIME_WAIT   -
tcp6       0      0 :::8009                 :::*                    LISTEN      9206/jsvc
tcp6       0      0 :::80                   :::*                    LISTEN      10730/apache2
tcp6       0      0 :::8180                 :::*                    LISTEN      9206/jsvc
tcp6       0      0 :::22                   :::*                    LISTEN      2431/sshd
tcp6       0      0 :::11100                :::*                    LISTEN      9133/java
tcp6       0      0 :::11101                :::*                    LISTEN      9160/java
tcp6       0      0 127.0.0.1:11100         127.0.0.1:38143         TIME_WAIT   -
tcp6       0      0 127.0.0.1:40725         127.0.0.1:11002         TIME_WAIT   -
tcp6       0      0 127.0.0.1:11101         127.0.0.1:54664         TIME_WAIT   -
tcp6       1      0 127.0.0.1:8009          127.0.0.1:35156         CLOSE_WAIT  9206/jsvc
tcp6       0      0 212.85.38.148:80        62.99.208.50:49516      TIME_WAIT   -
tcp6       0      0 212.85.38.148:80        62.155.235.89:50654     TIME_WAIT   -
tcp6       0      0 127.0.0.1:11101         127.0.0.1:54649         TIME_WAIT   -
tcp6       1      0 127.0.0.1:8009          127.0.0.1:33159         CLOSE_WAIT  9206/jsvc
tcp6       1      0 127.0.0.1:8009          127.0.0.1:60682         CLOSE_WAIT  9206/jsvc
tcp6       1      0 127.0.0.1:8009          127.0.0.1:33138         CLOSE_WAIT  9206/jsvc
tcp6       0      0 127.0.0.1:40743         127.0.0.1:11002         TIME_WAIT   -
tcp6       0      0 127.0.0.1:40733         127.0.0.1:11002         TIME_WAIT   -
tcp6       0      0 127.0.0.1:40740         127.0.0.1:11002         TIME_WAIT   -
tcp6       0      0 127.0.0.1:11100         127.0.0.1:38136         TIME_WAIT   -
tcp6       0      0 127.0.0.1:11101         127.0.0.1:54656         TIME_WAIT   -
tcp6       0      0 127.0.0.1:11101         127.0.0.1:54652         TIME_WAIT   -
tcp6       0      0 127.0.0.1:11100         127.0.0.1:50778         TIME_WAIT   -
tcp6       0      0 127.0.0.1:40732         127.0.0.1:11002         TIME_WAIT   -
tcp6       0      0 127.0.0.1:40742         127.0.0.1:11002         TIME_WAIT   -
tcp6       0      0 127.0.0.1:11101         127.0.0.1:54667         TIME_WAIT   -
tcp6       0      0 127.0.0.1:40729         127.0.0.1:11002         TIME_WAIT   -
tcp6       0      0 127.0.0.1:40736         127.0.0.1:11002         TIME_WAIT   -
tcp6       0      0 127.0.0.1:11100         127.0.0.1:38140         TIME_WAIT   -
vovm1:~#

(The "grep -v ESTAB" is to eliminate the "ESTABLISHED" connections, which are maybe 
irrelevant here).

The "jsvc" process here is the wrapper which wraps the JVM and Tomcat to allow them to use

ports below 1024 and still run as non-root.  For practical purposes, consider it as "tomcat".

As you can see, there are some AJP connections (local port 8009) in the CLOSE_WAIT state 
also.  It is a normal state of a TCP connection.
What is less normal, is that they would remain in that state for a long time.  That means

that one of the sides of the TCP connection is not doing what it should do.

In my experience, under Linux, this can become a problem if you have hundreds of these.
At some point, the TCP stack becomes unresponsive and does not allow any new connection.

Sometimes, it can happen because the "client" side in the connection (the one who 
initially creates the connection to a server's "listening" socket), discards a Java object

which contains an open low-level OS socket object with a connection still open, without 
explicitly "closing" this connection first.
The discarded (and unreachable) Java object is then still sitting in the heap for a while,

and only a GC will actually destroy it and as a side-effect close the underlying connection.

That is why I was asking you to force a GC, to see if this made your CLOSE_WAIT 
connections disappear.  Apparently in your case it doesn't, which mean that yours is 
another case.

Now, instead of restarting Tomcat or doing a GC, have you tried to restart Apache ?
or maybe just do a /etc/init.d/apache2 reload.  That may kill off and restart the Apache 
children processes, and clean up their mod_jk connections to Tomcat.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message