From dev-return-81871-apmail-tomcat-dev-archive=tomcat.apache.org@tomcat.apache.org Sat Jun 16 00:13:35 2007 Return-Path: Delivered-To: apmail-tomcat-dev-archive@www.apache.org Received: (qmail 6045 invoked from network); 16 Jun 2007 00:13:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 16 Jun 2007 00:13:33 -0000 Received: (qmail 66895 invoked by uid 500); 16 Jun 2007 00:13:32 -0000 Delivered-To: apmail-tomcat-dev-archive@tomcat.apache.org Received: (qmail 66830 invoked by uid 500); 16 Jun 2007 00:13:32 -0000 Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Developers List" Delivered-To: mailing list dev@tomcat.apache.org Received: (qmail 66819 invoked by uid 99); 16 Jun 2007 00:13:32 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Jun 2007 17:13:32 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [12.11.148.122] (HELO relay2.ptc.com) (12.11.148.122) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Jun 2007 17:13:26 -0700 X-IronPort-AV: E=Sophos;i="4.16,427,1175486400"; d="scan'208";a="234827490" Received: from hq-ex3fe2.ptcnet.ptc.com ([132.253.201.63]) by relay2.ptc.com with ESMTP; 15 Jun 2007 20:13:04 -0400 Received: from [132.253.40.50] ([132.253.40.50]) by HQ-EX3FE2.ptcnet.ptc.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 15 Jun 2007 20:13:03 -0400 Message-ID: <46732B5E.4090007@ptc.com> Date: Fri, 15 Jun 2007 19:14:22 -0500 From: Jess Holle User-Agent: Thunderbird 2.0.0.4 (Windows/20070604) MIME-Version: 1.0 To: Tomcat Developers List Subject: Re: Serious non-native AJP connector issue References: <200706142252.l5EMqPYj021740@maui.wilshire.com> <4671E278.8020204@ptc.com> <4671FC01.60408@ptc.com> <46727387.3040803@ptc.com> <46728BB3.7070307@ptc.com> <4672EA8B.10707@kippdata.de> In-Reply-To: <4672EA8B.10707@kippdata.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 16 Jun 2007 00:13:03.0666 (UTC) FILETIME=[1B5C7120:01C7AFAB] X-Virus-Checked: Checked by ClamAV on apache.org Rainer Jung wrote: > Jess, > > I didn't really carefully think about the case of a saturated pool. > But nevertheless some hints: > > 1) There is always one thread waiting for the accept, which is a usual > pool thread. So an offset of one between threeads processing requests > and the pool size is normal. Got that, but I've accounted for that via maxThreads of 51 and BalanceMember max of 50. I'm left wondering why there's an off-by-one beyond this (and thus an off-by-2 overall). > 2) There is no orderly shutdown in AJP13 for a connection, neither > from the httpd side, not from the Tomcat side. mod_jk will realize > closed connections and reopen them, but I have the impression, that > mod_proxy fails a backend when it gets a connection error. > > Tomcat in my experience reliably closes a connection and frees the > thread, when the web server does. > > Most of the time it makes sense to have a connectionTimeout > (Milliseconds) and a connection_pool_timeout (seconds, mod_jk) resp. > ttl (seconds, mod_proxy) which are in sync. But: they will never > harmonize completely, because mod_jk only checks for the need to throw > away closed connections during maintenance (every 60 seconds or > whatever is configured with worker.maintain) and I think mod_proxy > checks the ttl whenever a connection is put back into the pool. I don't think any of those should be involved in this short test. > 3) Does it happen with mod_jk too? I believe so. I did some mod_jk testing to verify the overall nature of the issue remained the same, but I didn't go through all the same detailed tests. I could redo this particular test if that's helpful. > 4) Weird guesses: > > - max is limited with mod_proxy to the number of threads per process > configured in your MPM (worker?). > > This is 25 by default. So if we want to have an easy to understand > scenario, set your threads per process to say 60 and min=max=50 as > well as removing the smax and the ttl.. That way 50 connections should > be started on startup (per httpd process; check with netstat), and we > shouldn't see any resizing during your ab test. Now start your ab test > before Tomcat will timeout (I remember that connectionTimeout had some > default value, even if you don't set it, but it is in the minutes). My Apache MaxClients is 300 and this is on Windows so I only have 1 worker process. > If you don't run into trouble then, we know your observation has to do > with resizing of connection pools. > > *Maybe*: ab is too fast and can come back with new requests faster, > than the connections go back into the pool after the end of a request > in httpd. Not very reasonable, but possible. Of cource the pool is > synchronized and the lock doesn't need to behave fair, i.e. when it > gets contended, it's not clear if the oldest waiting thread gets it > first. I believe I disproved this at some point by running 2 tests with -n 50 and -c 50 manually back to back, but I'd have to re-test to be sure. [I'm wishing I'd taken better notes of various results...] Apart from this weird edge condition (an off-by-2 isn't that devastating if it stays "2" in all cases), the thing that gets me overall is that the documentation really makes it sound like "acceptCount" works like a fair queue and that there's no harm in exceeding maxThreads except that requests will be queued. As Bill suggested, I should come up with suggested patches to the documentation -- I'm just not yet confident enough in my understanding to propose such patches. At this point all I can propose is strong warning verbage around maxThreads and acceptCount regarding their behavior for the Java AJP connector. -- Jess Holle --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org For additional commands, e-mail: dev-help@tomcat.apache.org