Return-Path: Delivered-To: apmail-tomcat-users-archive@www.apache.org Received: (qmail 43251 invoked from network); 19 Jan 2007 13:20:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 19 Jan 2007 13:20:24 -0000 Received: (qmail 84226 invoked by uid 500); 19 Jan 2007 13:19:53 -0000 Delivered-To: apmail-tomcat-users-archive@tomcat.apache.org Received: (qmail 83962 invoked by uid 500); 19 Jan 2007 13:19:49 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 83318 invoked by uid 99); 19 Jan 2007 13:19:38 -0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (hermes.apache.org: domain of lists@nabble.com designates 72.21.53.35 as permitted sender) Received: from [72.21.53.35] (HELO talk.nabble.com) (72.21.53.35) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jan 2007 05:15:27 -0800 Received: from [72.21.53.38] (helo=jubjub.nabble.com) by talk.nabble.com with esmtp (Exim 4.50) id 1H7tZu-0008Kc-OZ for users@tomcat.apache.org; Fri, 19 Jan 2007 05:14:46 -0800 Message-ID: <8448796.post@talk.nabble.com> Date: Fri, 19 Jan 2007 05:14:46 -0800 (PST) From: JNeuhoff To: users@tomcat.apache.org Subject: Re: Apache mod_jk memory leak? In-Reply-To: <45AFEBF8.3070800@kippdata.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: neuhoff@mhccorp.com References: <8398261.post@talk.nabble.com> <45ADDC28.9010206@kippdata.de> <8407489.post@talk.nabble.com> <45ADFA20.3000500@kippdata.de> <8410578.post@talk.nabble.com> <45AF353E.7070309@kippdata.de> <8428869.post@talk.nabble.com> <45AF5CAB.3080905@kippdata.de> <8436127.post@talk.nabble.com> <45AFEBF8.3070800@kippdata.de> X-Virus-Checked: Checked by ClamAV on apache.org I have just managed to repeat the error. 2 of us, from 2 different workstations, hammered our website for a minute, by rapidly clicking on links within the same site before it ended up always responding with a standard Error 503 (Service unavailable) coming from the Apache frontend. I checked all the log files, and again the only errors recorded were found in the mod_jk.log: .... [Fri Jan 19 11:34:42 2007] [0404:2392] [info] mod_jk.c (2142): Service error=0 for worker=ajp13 [Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (948): (ajp13) can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1566): (ajp13) Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1877): (ajp13) receiving from tomcat failed, recoverable operation attempt=1 [Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1916): (ajp13) sending request to tomcat failed, recoverable operation attempt=2 [Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1928): (ajp13) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port .... The 2 connections between Apache and Tomcat according the the netstat -a command just after the web server started the 503 (Service unavilable) responses (hightlighted in bold): Proto Local Address Foreign Address State TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:2093 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT TCP DAOHPRW2:2110 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED TCP DAOHPRW2:2077 daohprp2.essex.ac.uk:microsoft-ds TIME_WAIT TCP DAOHPRW2:2176 sernt29.essex.ac.uk:epmap ESTABLISHED TCP DAOHPRW2:2177 sernt29.essex.ac.uk:1025 ESTABLISHED TCP DAOHPRW2:2181 sernt29.essex.ac.uk:microsoft-ds TIME_WAIT TCP DAOHPRW2:2194 sernt50.essex.ac.uk:epmap ESTABLISHED TCP DAOHPRW2:2204 sernt5.essex.ac.uk:epmap ESTABLISHED TCP DAOHPRW2:2215 sernt25.essex.ac.uk:epmap ESTABLISHED TCP DAOHPRW2:2230 dalinux6a.essex.ac.uk:netbios-ssn ESTABLISHED TCP DAOHPRW2:2250 sernt2.essex.ac.uk:netbios-ssn TIME_WAIT TCP DAOHPRW2:2253 bodger.essex.ac.uk:microsoft-ds TIME_WAIT TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:http daohpr1.essex.ac.uk:1528 TIME_WAIT TCP DAOHPRW2:http daohpr2.essex.ac.uk:1936 TIME_WAIT TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:2232 sernt23.essex.ac.uk:netbios-ssn TIME_WAIT TCP DAOHPRW2:2237 sernt9.essex.ac.uk:netbios-ssn TIME_WAIT TCP DAOHPRW2:2245 serlx08.essex.ac.uk:netbios-ssn TIME_WAIT TCP DAOHPRW2:2249 sernt2.essex.ac.uk:netbios-ssn TIME_WAIT TCP DAOHPRW2:2261 sernt23.essex.ac.uk:netbios-ssn TIME_WAIT TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED Half another later, long after the connection_pool_timeout, connect_timeout and prepost_timeout, the netstat -a command still shows this: Proto Local Address Foreign Address State TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED TCP DAOHPRW2:2754 sernt29.essex.ac.uk:microsoft-ds TIME_WAIT TCP DAOHPRW2:2782 sernt29.essex.ac.uk:epmap ESTABLISHED TCP DAOHPRW2:2783 sernt29.essex.ac.uk:1025 ESTABLISHED TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING TCP DAOHPRW2:microsoft-ds daohpr1.essex.ac.uk:1541 ESTABLISHED TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED Yet if I then try again to access the web service running on Tomcat, mod_jk fails to re-establish a connection, according to the mod_jk.log: [Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (948): (ajp13) can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (1566): (ajp13) Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1877): (ajp13) receiving from tomcat failed, recoverable operation attempt=0 [Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1916): (ajp13) sending request to tomcat failed, recoverable operation attempt=1 [Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (948): (ajp13) can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1566): (ajp13) Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1877): (ajp13) receiving from tomcat failed, recoverable operation attempt=1 [Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1916): (ajp13) sending request to tomcat failed, recoverable operation attempt=2 [Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1928): (ajp13) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port [Fri Jan 19 12:50:52 2007] ajp13 daohprw2-a 0.281333 GET /ohpr/servlet/ HTTP/1.1 200 [Fri Jan 19 12:50:52 2007] [0404:2404] [info] mod_jk.c (2142): Service error=0 for worker=ajp13 Only Apache continues to respond with a standard 503 error according to its access.log: daohpr1.essex.ac.uk - - [19/Jan/2007:12:50:51 +0000] "GET /ohpr/servlet/ HTTP/1.1" 503 1101 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9" In short: There were only 2 established connections, Tomcat was running all the time, and, as expected, only saw 2 active HTTPSessions, yet the rapid hammering on weblinks within the website somehow blocked the connections between Apache and Tomcat. And now Tomcat keeps just listening on port 8009, but mod_jk is unable to establish any TCP connections between Apache and Tomcat. Only a complete server machine re-boot gets things back to normal. Any ideas what might have gone wrong? I am still using this workers.properties: # Define 1 real worker using ajp13 worker.list=ajp13 # Set properties for worker1 (ajp13) worker.ajp13.type=ajp13 worker.ajp13.host=localhost worker.ajp13.port=8009 worker.ajp13.connection_pool_timeout=600 worker.ajp13.connection_pool_minsize=10 worker.ajp13.connect_timeout=15000 worker.ajp13.prepost_timeout=10000 Regards Juergen Neuhoff -- View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8448796 Sent from the Tomcat - User mailing list archive at Nabble.com. --------------------------------------------------------------------- To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org