tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bill Barker" <>
Subject Re: Serious non-native AJP connector issue
Date Fri, 15 Jun 2007 01:48:12 GMT

"Jess Holle" <> wrote in message
> There's no intent to handle this case?  Really?
> Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount'
> is set to anything other than 0 explaining this if this is the case!
> If so, this is very unfortunate.  We use "maxThreads" as a throttle to
> limit the concurrency at this level under the (silly?) assumption that
> "acceptCount" behaves as documented in the documentation.  [Yes, one
> could argue that a separate throttle should be used behind this layer,
> which we might have been inclined to do if the documentation said
> "acceptCount does not work".]

You are misunderstanding how AJP works.  Since you don't have a 
connectionTimeout on the <Connector />, the connections to httpd stay alive 
waiting to get another request on the same socket.  As a result, there won't 
be any free Threads to handle a new connection so it doesn't matter what 
acceptCount is.  Their isn't anyone there to accept them.

> Further, I've been able to reproduce such issues using a concurrency of
> exactly the "maxThreads".  At *best* there is an off-by-one error here,
> but I've still seen problems in some testing using a concurrency level 2
> or 3 less than "maxThreads" -- which leads me to believe there is
> something a bit more nefarious going on here.  Another of my colleagues
> claims that he's encountered issues in simple ab tests quite a ways
> below "maxThreads".  I've not managed to reproduce this case myself, but
> I could certainly focus on this case if the Tomcat contributors have no
> intent to fix "acceptCount" behavior for the Java AJP connector.

The thread waiting to accept new connections counts as a thread for the 
purposes of maxThreads, so yes it is one less.  And, I can save you the 
trouble of investigating, since as long as your maxThreads <= pool size in 
httpd you will see this sort of thing.

> Is there a firm intent to handle this case in the _/native/_ AJP
> connector?  If so, then this distinction should be clearly stated (and
> an IllegalArgumentException thrown as per above when the native
> connector cannot be found/initialized).

The APR/AJP connector (as well as the experimental NIO/AJP connector) use a 
many-to-one connection-to-thread setup so you can have a large number of 
connections sitting and doing nothing and use relatively few threads.  With 
the default Java connector the mapping is one-to-one.

> Personally I see utility in having "acceptCount" working in the AJP
> case.  If one exceeds the mod_proxy_ajp hard maximum the client
> immediately gets a 503.  There are cases when just queuing a few
> requests makes more sense than either returning 503's or letting them
> rip -- which is where acceptCount comes in.  [Granted the acceptCount
> numbers below are excessive, but that's another matter...]

It's not that acceptCount doesn't work, it's that without a 
connectionTimeout each thread waits forever listening on it's connection for 
the next request to come in.  With a connectionTimeout, once httpd hasn't 
sent another request on this connection after the timeout, the thread is 
recycled and can handle a new connection.

> --
> Jess Holle
> Bill Barker wrote:
>> Yes, since you know in advance how many connections you are going to get 
>> to
>> the AJP connector, you can configure it so that it has enough threads to
>> handle all of those connections.  That is why it doesn't attempt to 
>> handle
>> the case when the concurrency goes above maxThreads
>>> -----Original Message-----
>>> From: Jess Holle []
>>> Sent: Thursday, June 14, 2007 2:58 PM
>>> To: Tomcat Developers List
>>> Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
>>> Subject: Serious non-native AJP connector issue
>>> We're facing a /serious /issue with the non-native AJP connector.
>>> To put it most simply, some requests seem to "get lost" in Tomcat in
>>> various cases involving concurrent requests -- and not
>>> egregious numbers
>>> of concurrent requests, either.
>>> For instance,
>>>    1. Use a Tomcat 5.5.23 with a configuration like:
>>>         <Connector port="8010"
>>>                    minSpareThreads="4" maxSpareThreads="12"
>>>     maxThreads="18" acceptCount="282"
>>>                    tomcatAuthentication="false"
>>>     useBodyEncodingForURI="true" URIEncoding="UTF-8"
>>>                    enableLookups="false" redirectPort="8443"
>>>     protocol="AJP/1.3" />
>>>     (which are intended solely for making it easier to test
>>> concurrency
>>>     issues that to overrun a realistic 'maxThreads' setting) and as a
>>>     control, similar thread pool settings on the direct HTTP
>>> connector:
>>>         <Connector port="8080" maxHttpHeaderSize="8192"
>>>                    minSpareThreads="4" maxSpareThreads="12"
>>>     maxThreads="18" acceptCount="282"
>>>                    enableLookups="false" redirectPort="8443"
>>>                    connectionTimeout="20000"
>>> disableUploadTimeout="true" />
>>>    2. Use an Apache 2.2.4 with mod_proxy_ajp with a
>>> configuration like:
>>>         <Proxy balancer://ajpWorker>
>>>         BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
>>>         ttl=900 keepalive=Off timeout=900
>>>         </Proxy>
>>>         RewriteEngine on
>>>         RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
>>>         balancer://ajpWorker$1 [P]
>>>     (on Windows in this case; similar results can be obtained on Linux
>>>     at least)
>>>    3. Use a simple test JSP page (placed in a web app
>>> containing nothing
>>>       else):
>>>         <%@page session="false"
>>>         %><%@page contentType="text/html" pageEncoding="UTF-8"
>>>         %><%!
>>>           private static final String  titleString = "Sleepy
>>> Test JSP Page";
>>>         %><html>
>>>         <head>
>>>         <%
>>>           String  sleepSeconds = request.getParameter( "secs" );
>>>           if ( sleepSeconds == null )
>>>             sleepSeconds = "1";
>>>           long  secsToSleep = Long.parseLong( sleepSeconds );
>>>           Thread.sleep( 1000L * secsToSleep );
>>>         %>
>>>         <meta http-equiv="Content-Type" content="text/html;
>>> charset=UTF-8"/>
>>>         <title><%=titleString%></title>
>>>         </head>
>>>         <body>
>>>         <center>
>>>         <h2><%=titleString%>: SUCCESS!</h2>
>>>         [Slept <%= secsToSleep %> seconds.]
>>>         </center>
>>>         </body>
>>>         </html>
>>>    4. Hit the page with ab
>>>           * First, test direct Tomcat connections:
>>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>>                   http://hostname:*8080*/TestApp/test.jsp?secs=3
>>>                       + Result: All requests complete sucessfully in
>>>                         6118 ms.
>>>           * Second, test connections via Apache:
>>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>>                   http://hostname/TestApp/test.jsp?secs=3
>>>                       + Result: Only 17 requests complete before ab
>>>                         times out.
>>>           * Third, test connections via Apache again soon (under the
>>>             BalancerMember 'timeout' seconds) after the last test
>>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>>                   http://hostname/TestApp/test.jsp?secs=3
>>>                       + Result: Only 9 requests complete
>>> before ab times
>>>                         out.
>>> Something is clearly /horribly/ wrong with the handling of any
>>> concurrency over 'maxThreads' in this case.  Even so, there
>>> seems to be
>>> some sort of "off-by-one" error in that only 17 requests
>>> complete, not
>>> 18.  Worse, this has a lingering effect that decreases
>>> Tomcat's ability
>>> to concurrent requests thereafter (with this impact seemingly
>>> being much
>>> worse the longer the BalancerMember timeout is set to -- and we have
>>> some very long running requests and thus need this timeout to
>>> be /very/
>>> large).
>>> This is not the only ill effect we've seen when hitting Tomcat 5.5.24
>>> with concurrent requests in this manner, but it is a good
>>> place to start.
>>> As for the native connector, it just works here.  So why don't I just
>>> use it?  Well, first off, we have to support Tomcat on
>>> Windows (32 and
>>> 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX.  We've
>>> been unable to get the connector built on some of these
>>> platforms and on
>>> some we can't get the resulting binary to function (specifically on
>>> AIX).  Further, we had some stability issues with the native
>>> connector
>>> in the past and had considered the Java connector the safest, if not
>>> fastest, option -- and to a degree given that everything is
>>> Java I still
>>> feel that's the case.  Finally, however, this connector should just
>>> plain work.  Tomcat shouldn't be a cripple unless/until you manage to
>>> build a native connector for your platform.
>>> Any troubleshooting and/or debugging ideas (e.g. where
>>> exactly to place
>>> breakpoints, what logs to turn on, etc, etc) would be
>>> /greatly/ appreciated.
>>> --
>>> Jess Holle
>>> P.S. If this should go to the user's mailing list instead
>>> that's fine,
>>> but this really seemed like a developer-oriented issue to me.
>> This message is intended only for the use of the person(s) listed above 
>> as the intended recipient(s), and may contain information that is 
>> PRIVILEGED and CONFIDENTIAL.  If you are not an intended recipient, you 
>> may not read, copy, or distribute this message or any attachment. If you 
>> received this communication in error, please notify us immediately by 
>> e-mail and then delete all copies of this message and any attachments.
>> In addition you should be aware that ordinary (unencrypted) e-mail sent 
>> through the Internet is not secure. Do not send confidential or sensitive 
>> information, such as social security numbers, account numbers, personal 
>> identification numbers and passwords, to us via ordinary (unencrypted) 
>> e-mail.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message