httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Murray" <ryan.mur...@elementn.com>
Subject RE: [users@httpd] mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does not understand this error code: proxy: read response failed
Date Mon, 02 Nov 2009 10:12:37 GMT

-----Original Message-----
From: Damian Traverso [mailto:damian.traverso@vostu.com] 
Sent: Sunday, November 01, 2009 4:59 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does not understand
this error code: proxy: read response failed

Hi, first of all, thanks for your answer.

And about the error, I also thought that could be the problem, but the
thing is when that error is logged on Apache, the Tomcat instances are
NOT using their MaxThreads capabilities.

Another issue we have is that, when this problem appears, the
processing threads on Tomcat start taking longer responses than
regular usage.

Thanks again,
-Damian

On Sat, Oct 31, 2009 at 7:46 PM, André Warnier <aw@ice-sa.com> wrote:
> Damian Traverso wrote:
>>
>> Hi,
>> we have one Apache-2.2.13 running mod_proxy_ajp + mod_proxy_balancer,
>> connected to (3) Tomcat-6.0.20 instances under Fedora release 8. We
>> are experiencing some issues with high CPU load on the Tomcat side,
>> and Apache starts logging errors like these
>>
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>> [Fri Oct 30 14:47:43 2009] [error] ajp_read_header: ajp_ilink_receive
>> failed
>> [Fri Oct 30 14:47:43 2009] [error] (120006)APR does not understand
>> this error code: proxy: read response failed from
>>
>> On Tomcat side we are using the APR Native Connector, with this settings:
>>
>>  <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
>> maxThreads="130" backlog="10" connectionTimeout="30000"
>> keepAliveTimeout="30000" enableLookups="false" />
>>
>> Apache MPM conf:
>>
>> ServerLimit 100
>> <IfModule mpm_worker_module>
>>   StartServers         30
>>   MaxClients         2500
>>   MinSpareThreads      50
>>   MaxSpareThreads     100
>>   ThreadsPerChild      25
>>   MaxRequestsPerChild   0
>> </IfModule>
>>
>> mod_proxy_ajp conf:
>>
>> <Proxy balancer://jogacraque>
>>   BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app01 ttl=30 max=2
>>   BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app02 ttl=30 max=2
>>   BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app03 ttl=30 max=2
>> </Proxy>
>>
>> Apache specs:
>>
>> Server version: Apache/2.2.13 (Unix)
>> Server built:   Sep 29 2009 11:12:35
>> Server's Module Magic Number: 20051115:23
>> Server loaded:  APR 1.3.8, APR-Util 1.3.9
>> Compiled using: APR 1.3.8, APR-Util 1.3.9
>> Architecture:   32-bit
>> Server MPM:     Worker
>>  threaded:     yes (fixed thread count)
>>  forked:     yes (variable process count)
>>
>>
>> I've beeng testing different settings for mod_proxy_ajp, but I am not
>> sure what is the exact relation between the max option for mod_proxy
>> and ThreadsPerChild. Should I set max to 1? I understand I would have
>> as many connections to each Tomcat box as
>> max*(MaxClients/ThreadsPerChild), is this correct?
>>
>> BTW, is the total number of connections that mod_proxy_ajp will
>> forward to Tomcat related with MaxThreads ? I don't fully understand
>> how APR connector works in this case.
>>
> Hi.
> No great specialist here, but I'll go by the numbers above and give you my
> interpretation.
>
> MaxClients 2500
> on your webserver/balancer indicates how many requests you are willing to
> accept and process at the same time, at the Apache level.
> ThreadsPerChild 25
> means that, if you ever reach these 2500 simultaneous requests, Apache will
> create 100 children processes to handle them, since each child is limited to
> 25 threads, and 2500 / 25 threads -> 100 children.
> (Which seems to match the ServerLimit. Good.)
>
> It is so with the mod_jk connector, and probably also with mod_proy_ajp,
> that the connector will automatically calculate how many connections to make
> to Tomcat, based on the number of threads/processes above.
>
> At the Tomcat level, the AJP Connector says
> maxThreads="130"
> and you have 3 Tomcats, thus for a total maximum number of threads of 130 X
> 3 = 390.
> One Tomcat thread processes one request at a time.
> So the Tomcats, together, can process 390 requests at a time.
>
> There seems to be a mismatch between the number of requests you accept in
> Apache (2500), and the number of threads available to process them in Tomcat
> (390). At the limit, you could have 390 requests being processed in Tomcat,
> and 2110 waiting to get a free Tomcat thread.
>
> That is probably what you are seeing with this log message :
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>
> In other words, probably Apache/mod_proy_ajp got tired of waiting.
>
> On the other hand, you indicate that the poor Tomcats are already struggling
> to cope with the load.  Increasing their number of threads is thus unlikely
> to help much.
> So it looks like you need more Tomcats.
> Or else reduce the MaxClients in Apache.
> Or optimise your Tomcat webapps.
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>  "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Hi Damian,

I've have experienced these errors for years over many versions of tomcat and APR/mod_ajp,
always without a full load on Tomcat.  I suspect it's a native socket management issue in
the APR lib.  I have noticed more errors on some OSs than others.  But I'm no expert either.

What I do recommend is to stop using AJP and use mod_proxy_http instead.  Unless you need
the high degree of tunability of the AJP connectors or the SSL pass through features, IMHO
HTTP proxying will give you less headaches and great performance under most loads.  I used
to have frequent errors, delays and even hanging servers with AJP which I could never diagnose
effectively, but have no problems using HTTP proxying.

Also, if you apaches are serving your static content, then configuring more workers at the
apache level than Tomcat threads is normal.  In fact, if you are serving all your static content
from Apache, depending on your hardware, then you may already have too _many_ threads configured
on Tomcat.  If there are some longer running requests to process which use the app server
CPU intensively, you could easily swamp you Tomcat trying to process 130 simultaneous requests.
 If you've already load tested and this is the right setup for you workload then great.

Cheers,
Ryan


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message