tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Mikusa <dmik...@vmware.com>
Subject Re: ajp_ilink_receive error - please advise
Date Fri, 21 Sep 2012 21:00:40 GMT
On Sep 20, 2012, at 6:12 PM, Django Radonich-Camp wrote:

> thank you for the replies chris and dan.
> 
> chris:
> 1.  upgrading apache is possible, but not easy (puppet manages the install).
> 2.  unfortunately i have not figured out how to reproduce the problem.  we
> did extensive load testing on the system prior to launch and not once did
> we see this problem...
> 
> dan:
> 1.  good idea.  do you know a good reference for setting this up?

It is just a matter of adding the JVM options in setenv.sh.

I'll typically use something like this...

-XX:+PrintGC -Xloggc:$CATALINA_BASE/logs/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

You can adjust the options to fit your needs.  If you need more detail, I would suggest google'ing
"java garbage collection log" or the individual option.

> 2.  also, what is the process for capturing thread dumps?  this will be
> difficult however because the incidents are random and so far, mostly
> ocurring in the middle of the night.

There are a few ways to do this.  I find running "kill -3 <pid>" to be the easiest.
 You could also use "jstack <pid>"

Dan


> 
> thanks again for the help.
> 
> 
> On Thu, Sep 20, 2012 at 5:35 AM, Daniel Mikusa <dmikusa@vmware.com> wrote:
> 
>> On Sep 19, 2012, at 5:38 PM, Django Radonich-Camp wrote:
>> 
>>> hello.
>>> 
>>> we are running an application on tomcat and experiencing intermittent
>>> periods where the application is non-responsive and thus non-functional.
>>> the general set up is apache and tomcat, with mod_proxy_ajp as the
>>> connector (specific details and configs below).
>>> 
>>> during these events, the primary apache error log shows the following:
>>> 1. (104)Connection reset by peer: ajp_ilink_receive() can't receive
>> header
>>> 2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
>>> receive header
>>> 
>>> it looks like error #1 repeats for a while and then error #2 repeats for
>> a
>>> while.
>>> 
>>> at the same time in the application specific apache error log (as
>> specified
>>> in the vhost set up for app) we see the following errors repeated (though
>>> in mixed order from below):
>>> 3. [error] ajp_read_header: ajp_ilink_receive failed
>>> 4. [error] (120006)APR does not understand this error code: proxy: read
>>> response failed from (null) (localhost)
>>> 5. [error] (120006)APR does not understand this error code: proxy: read
>>> response failed from 127.0.0.1:8009 (localhost)
>>> 6. [error] [client X.X.X.X] proxy: error processing end
>>> 
>>> and occasionally:
>>> 7. [error] proxy: read zero bytes, expecting 464 bytes
>>> 
>>> the catalina.out log registers nothing during the time period the
>>> application is unresponsive.
>>> 
>>> a couple of other things to note:
>>> - these events are coming under light to no load as far as i can tell.
>>> - these events last from about 5 to 30 minutes and then everything works
>>> again as expected with no manual intervention.
>>> - the time of day of for the events is not consistent.
>>> - these events initially occured rarely, but over the last month have
>>> ramped up to daily.
>>> 
>>> any suggestions on how to fix or further troubleshoot this problem? and
>>> thanks in advance for the help.
>> 
>> Here are a couple additional troubleshooting steps you can take on your
>> Tomcat instances...
>> 
>> 1.) Enable garbage collection logging.  Look for any full GC's.
>> 2.) Take some thread dumps during an incident.  Look for blocking.
>> 
>> Dan
>> 
>> 
>>> 
>>> below please find more information on versions and configs... let me know
>>> if more info is needed.
>>> 
>>> OS Name:        Linux (ubuntu 10.04.4)
>>> OS Version:     2.6.32-31-server
>>> Architecture:   amd64
>>> JVM Version:    1.6.0_32-b05
>>> JVM Vendor:     Sun Microsystems Inc.
>>> Server version: Apache Tomcat/6.0.24
>>> apache: Apache/2.2.14
>>> 
>>> MPM configs
>>> --------START--------------
>>> <IfModule mpm_prefork_module>
>>>  StartServers          5
>>>  MinSpareServers       5
>>>  MaxSpareServers      10
>>>  MaxClients           26
>>>  MaxRequestsPerChild  1000
>>> </IfModule>
>>> --------END----------------
>>> 
>>> balancer conf
>>> --------START--------------
>>> <Proxy balancer://mysite_balancer*>
>>>   Order deny,allow
>>>   Allow from all
>>> </Proxy>
>>> 
>>> ProxyPassMatch ^/(.+.cf[cm])(.*)?$
>> balancer://mysite_balancer/irised/client
>>> stickysession=JSESSIONID|jsessionid
>>> 
>>> ProxyPassReverseCookiePath /irised /
>>> 
>>> <Proxy balancer://mysite_balancer>
>>>   BalancerMember ajp://localhost:8009 route=www1 retry=5
>>> </Proxy>
>>> --------END----------------
>>> 
>>> connector xml
>>> --------START--------------
>>> <Connector URIEncoding="UTF-8"
>>>          port="8009"
>>>          protocol="AJP/1.3"
>>>          connectionTimeout="20000"
>>>          redirectPort="8443" />
>>> --------END----------------
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message