tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JNeuhoff <neuh...@mhccorp.com>
Subject Re: Apache mod_jk memory leak?
Date Fri, 19 Jan 2007 13:14:46 GMT

I have just managed to repeat the error. 2 of us, from 2 different
workstations, hammered our website for a minute, by rapidly clicking on
links within the same site before it ended up always responding with a
standard Error 503 (Service unavailable) coming from the Apache frontend. I
checked all the log files, and again the only errors recorded were found in
the mod_jk.log:

....
[Fri Jan 19 11:34:42 2007] [0404:2392] [info]  mod_jk.c (2142): Service
error=0 for worker=ajp13
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 11:34:42 2007] [0404:2388] [info]  jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=1
[Fri Jan 19 11:34:42 2007] [0404:2388] [info]  jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed,  recoverable operation attempt=2
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1928):
(ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
listening on the wrong port
....


The 2 connections between Apache and Tomcat according the the netstat -a
command just after the web server started the 503 (Service unavilable)
responses (hightlighted in bold):

  Proto  Local Address          Foreign Address        State
  TCP    DAOHPRW2:epmap         DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:microsoft-ds  DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:1078          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:3389          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:8009          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:8080          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:10000         DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:2093          DAOHPRW2.essex.ac.uk:8009  TIME_WAIT
  TCP    DAOHPRW2:2110          DAOHPRW2.essex.ac.uk:8009  TIME_WAIT
  TCP    DAOHPRW2:8005          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:netbios-ssn   DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:2076          daohprp2.essex.ac.uk:3306  ESTABLISHED
  TCP    DAOHPRW2:2077          daohprp2.essex.ac.uk:microsoft-ds  TIME_WAIT
  TCP    DAOHPRW2:2176          sernt29.essex.ac.uk:epmap  ESTABLISHED
  TCP    DAOHPRW2:2177          sernt29.essex.ac.uk:1025  ESTABLISHED
  TCP    DAOHPRW2:2181          sernt29.essex.ac.uk:microsoft-ds  TIME_WAIT
  TCP    DAOHPRW2:2194          sernt50.essex.ac.uk:epmap  ESTABLISHED
  TCP    DAOHPRW2:2204          sernt5.essex.ac.uk:epmap  ESTABLISHED
  TCP    DAOHPRW2:2215          sernt25.essex.ac.uk:epmap  ESTABLISHED
  TCP    DAOHPRW2:2230          dalinux6a.essex.ac.uk:netbios-ssn 
ESTABLISHED
  TCP    DAOHPRW2:2250          sernt2.essex.ac.uk:netbios-ssn  TIME_WAIT
  TCP    DAOHPRW2:2253          bodger.essex.ac.uk:microsoft-ds  TIME_WAIT
  TCP    DAOHPRW2:http          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:http          daohpr1.essex.ac.uk:1528  TIME_WAIT
  TCP    DAOHPRW2:http          daohpr2.essex.ac.uk:1936  TIME_WAIT
  TCP    DAOHPRW2:netbios-ssn   DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:2232          sernt23.essex.ac.uk:netbios-ssn  TIME_WAIT
  TCP    DAOHPRW2:2237          sernt9.essex.ac.uk:netbios-ssn  TIME_WAIT
  TCP    DAOHPRW2:2245          serlx08.essex.ac.uk:netbios-ssn  TIME_WAIT
  TCP    DAOHPRW2:2249          sernt2.essex.ac.uk:netbios-ssn  TIME_WAIT
  TCP    DAOHPRW2:2261          sernt23.essex.ac.uk:netbios-ssn  TIME_WAIT
  TCP    DAOHPRW2:3389          daohpr1.essex.ac.uk:1533  ESTABLISHED


Half another later, long after the connection_pool_timeout, connect_timeout
and prepost_timeout, the netstat -a  command still shows this:

  Proto  Local Address          Foreign Address        State
  TCP    DAOHPRW2:epmap         DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:microsoft-ds  DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:1078          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:3389          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:8009          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:8080          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:10000         DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:8005          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:netbios-ssn   DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:2076          daohprp2.essex.ac.uk:3306  ESTABLISHED
  TCP    DAOHPRW2:2754          sernt29.essex.ac.uk:microsoft-ds  TIME_WAIT
  TCP    DAOHPRW2:2782          sernt29.essex.ac.uk:epmap  ESTABLISHED
  TCP    DAOHPRW2:2783          sernt29.essex.ac.uk:1025  ESTABLISHED
  TCP    DAOHPRW2:http          DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:netbios-ssn   DAOHPRW2.essex.ac.uk:0  LISTENING
  TCP    DAOHPRW2:microsoft-ds  daohpr1.essex.ac.uk:1541  ESTABLISHED
  TCP    DAOHPRW2:3389          daohpr1.essex.ac.uk:1533  ESTABLISHED


Yet if I then try again to access the web service running on Tomcat, mod_jk
fails to re-establish a connection, according to the mod_jk.log:

[Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 12:50:51 2007] [0404:2404] [info]  jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=0
[Fri Jan 19 12:50:51 2007] [0404:2404] [info]  jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed,  recoverable operation attempt=1
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 12:50:52 2007] [0404:2404] [info]  jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=1
[Fri Jan 19 12:50:52 2007] [0404:2404] [info]  jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed,  recoverable operation attempt=2
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1928):
(ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
listening on the wrong port
[Fri Jan 19 12:50:52 2007] ajp13 daohprw2-a 0.281333 GET /ohpr/servlet/
HTTP/1.1 200
[Fri Jan 19 12:50:52 2007] [0404:2404] [info]  mod_jk.c (2142): Service
error=0 for worker=ajp13

Only Apache continues to respond with a standard 503 error according to its
access.log:

daohpr1.essex.ac.uk - - [19/Jan/2007:12:50:51 +0000] "GET /ohpr/servlet/
HTTP/1.1" 503 1101 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"

In short: There were only 2 established connections, Tomcat was running all
the time, and, as expected, only saw 2 active HTTPSessions, yet the rapid
hammering on weblinks within the website somehow blocked the connections
between Apache and Tomcat. And now Tomcat keeps just listening on port 8009,
but mod_jk is unable to establish any TCP connections between Apache and
Tomcat. Only a complete server machine re-boot gets things back to normal.

Any ideas what might have gone wrong? I am still using this
workers.properties:

# Define 1 real worker using ajp13
worker.list=ajp13
# Set properties for worker1 (ajp13)
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
worker.ajp13.connection_pool_timeout=600
worker.ajp13.connection_pool_minsize=10
worker.ajp13.connect_timeout=15000
worker.ajp13.prepost_timeout=10000


Regards

Juergen Neuhoff

-- 
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8448796
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message