tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jess Holle <je...@ptc.com>
Subject Re: Serious non-native AJP connector issue
Date Fri, 15 Jun 2007 00:51:04 GMT
There's no intent to handle this case?  Really?

Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount' 
is set to anything other than 0 explaining this if this is the case!

If so, this is very unfortunate.  We use "maxThreads" as a throttle to 
limit the concurrency at this level under the (silly?) assumption that 
"acceptCount" behaves as documented in the documentation.  [Yes, one 
could argue that a separate throttle should be used behind this layer, 
which we might have been inclined to do if the documentation said 
"acceptCount does not work".]

Further, I've been able to reproduce such issues using a concurrency of 
exactly the "maxThreads".  At *best* there is an off-by-one error here, 
but I've still seen problems in some testing using a concurrency level 2 
or 3 less than "maxThreads" -- which leads me to believe there is 
something a bit more nefarious going on here.  Another of my colleagues 
claims that he's encountered issues in simple ab tests quite a ways 
below "maxThreads".  I've not managed to reproduce this case myself, but 
I could certainly focus on this case if the Tomcat contributors have no 
intent to fix "acceptCount" behavior for the Java AJP connector.

Is there a firm intent to handle this case in the _/native/_ AJP 
connector?  If so, then this distinction should be clearly stated (and 
an IllegalArgumentException thrown as per above when the native 
connector cannot be found/initialized).

Personally I see utility in having "acceptCount" working in the AJP 
case.  If one exceeds the mod_proxy_ajp hard maximum the client 
immediately gets a 503.  There are cases when just queuing a few 
requests makes more sense than either returning 503's or letting them 
rip -- which is where acceptCount comes in.  [Granted the acceptCount 
numbers below are excessive, but that's another matter...]

--
Jess Holle
jessh@ptc.com

Bill Barker wrote:
> Yes, since you know in advance how many connections you are going to get to
> the AJP connector, you can configure it so that it has enough threads to
> handle all of those connections.  That is why it doesn't attempt to handle
> the case when the concurrency goes above maxThreads
>> -----Original Message-----
>> From: Jess Holle [mailto:jessh@ptc.com] 
>> Sent: Thursday, June 14, 2007 2:58 PM
>> To: Tomcat Developers List
>> Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
>> Subject: Serious non-native AJP connector issue
>>
>> We're facing a /serious /issue with the non-native AJP connector.
>>
>> To put it most simply, some requests seem to "get lost" in Tomcat in 
>> various cases involving concurrent requests -- and not 
>> egregious numbers 
>> of concurrent requests, either.
>>
>> For instance,
>>
>>    1. Use a Tomcat 5.5.23 with a configuration like:
>>
>>         <Connector port="8010"
>>                    minSpareThreads="4" maxSpareThreads="12"
>>     maxThreads="18" acceptCount="282"
>>                    tomcatAuthentication="false"
>>     useBodyEncodingForURI="true" URIEncoding="UTF-8"
>>                    enableLookups="false" redirectPort="8443"
>>     protocol="AJP/1.3" />
>>
>>     (which are intended solely for making it easier to test 
>> concurrency
>>     issues that to overrun a realistic 'maxThreads' setting) and as a
>>     control, similar thread pool settings on the direct HTTP 
>> connector:
>>
>>         <Connector port="8080" maxHttpHeaderSize="8192"
>>                    minSpareThreads="4" maxSpareThreads="12"
>>     maxThreads="18" acceptCount="282"
>>                    enableLookups="false" redirectPort="8443"
>>                    connectionTimeout="20000" 
>> disableUploadTimeout="true" />
>>
>>    2. Use an Apache 2.2.4 with mod_proxy_ajp with a 
>> configuration like:
>>
>>         <Proxy balancer://ajpWorker>
>>         BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
>>         ttl=900 keepalive=Off timeout=900
>>         </Proxy>
>>
>>         RewriteEngine on
>>         RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
>>         balancer://ajpWorker$1 [P]
>>
>>     (on Windows in this case; similar results can be obtained on Linux
>>     at least)
>>
>>    3. Use a simple test JSP page (placed in a web app 
>> containing nothing
>>       else):
>>
>>         <%@page session="false"
>>         %><%@page contentType="text/html" pageEncoding="UTF-8"
>>         %><%!
>>           private static final String  titleString = "Sleepy 
>> Test JSP Page";
>>         %><html>
>>         <head>
>>         <%
>>           String  sleepSeconds = request.getParameter( "secs" );
>>           if ( sleepSeconds == null )
>>             sleepSeconds = "1";
>>           long  secsToSleep = Long.parseLong( sleepSeconds );
>>           Thread.sleep( 1000L * secsToSleep );
>>         %>
>>         <meta http-equiv="Content-Type" content="text/html; 
>> charset=UTF-8"/>
>>         <title><%=titleString%></title>
>>         </head>
>>         <body>
>>         <center>
>>         <h2><%=titleString%>: SUCCESS!</h2>
>>         [Slept <%= secsToSleep %> seconds.]
>>         </center>
>>         </body>
>>         </html>
>>
>>    4. Hit the page with ab
>>           * First, test direct Tomcat connections:
>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>                   http://hostname:*8080*/TestApp/test.jsp?secs=3
>>                       + Result: All requests complete sucessfully in
>>                         6118 ms.
>>           * Second, test connections via Apache:
>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>                   http://hostname/TestApp/test.jsp?secs=3
>>                       + Result: Only 17 requests complete before ab
>>                         times out.
>>           * Third, test connections via Apache again soon (under the
>>             BalancerMember 'timeout' seconds) after the last test
>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>                   http://hostname/TestApp/test.jsp?secs=3
>>                       + Result: Only 9 requests complete 
>> before ab times
>>                         out.
>>
>> Something is clearly /horribly/ wrong with the handling of any 
>> concurrency over 'maxThreads' in this case.  Even so, there 
>> seems to be 
>> some sort of "off-by-one" error in that only 17 requests 
>> complete, not 
>> 18.  Worse, this has a lingering effect that decreases 
>> Tomcat's ability 
>> to concurrent requests thereafter (with this impact seemingly 
>> being much 
>> worse the longer the BalancerMember timeout is set to -- and we have 
>> some very long running requests and thus need this timeout to 
>> be /very/ 
>> large).
>>
>> This is not the only ill effect we've seen when hitting Tomcat 5.5.24 
>> with concurrent requests in this manner, but it is a good 
>> place to start.
>>
>> As for the native connector, it just works here.  So why don't I just 
>> use it?  Well, first off, we have to support Tomcat on 
>> Windows (32 and 
>> 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX.  We've 
>> been unable to get the connector built on some of these 
>> platforms and on 
>> some we can't get the resulting binary to function (specifically on 
>> AIX).  Further, we had some stability issues with the native 
>> connector 
>> in the past and had considered the Java connector the safest, if not 
>> fastest, option -- and to a degree given that everything is 
>> Java I still 
>> feel that's the case.  Finally, however, this connector should just 
>> plain work.  Tomcat shouldn't be a cripple unless/until you manage to 
>> build a native connector for your platform.
>>
>> Any troubleshooting and/or debugging ideas (e.g. where 
>> exactly to place 
>> breakpoints, what logs to turn on, etc, etc) would be 
>> /greatly/ appreciated.
>>
>> --
>> Jess Holle
>> jessh@ptc.com
>>
>> P.S. If this should go to the user's mailing list instead 
>> that's fine, 
>> but this really seemed like a developer-oriented issue to me.
>>
>>
>>     
>
>
>
> This message is intended only for the use of the person(s) listed above as the intended
recipient(s), and may contain information that is PRIVILEGED and CONFIDENTIAL.  If you are
not an intended recipient, you may not read, copy, or distribute this message or any attachment.
If you received this communication in error, please notify us immediately by e-mail and then
delete all copies of this message and any attachments.
>
> In addition you should be aware that ordinary (unencrypted) e-mail sent through the Internet
is not secure. Do not send confidential or sensitive information, such as social security
numbers, account numbers, personal identification numbers and passwords, to us via ordinary
(unencrypted) e-mail.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>   


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message