tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <>
Subject Re: mod_jk Problems - - worker went to error state and dont recover
Date Wed, 20 Feb 2008 17:59:01 GMT
Ahmed Musa wrote:
> Hello,
> Wow -thank you very much Rainer for your very quick and informative answer.
> I will go to 1.2.26 and think about some "smoother" Values for reply_timeout and max_reply_timeouts.
> I will search for the requests which causes the Problems - becasue i still log the response
time in your mentioned way - but I am not sure that the Userrequests are responsible for the

One note: for Apache httpd 2.x %d is microseconds (there is no format 
for milliseconds), for Tomcat %D is milliseconds. As long as you are 
searching for the root cause, it might make sense to have both access 
logs active to check about duration differences.

> So one further question - does mod_jk itself checks if the Backend is reachable - without

No. Everything only works on top of user requests.

> When there are connections to the Backend - are they closed after the respone or are
the hold open for further requests.

In general hold open. There are parameters on how long they are held 
open without more requests before they get shut down, and also how many 
might be kept open even when no requests are coming in. Those are the 
connection pool parameters, which you will find on

Tomcat also has a connectionTimeout on the connector, which will shut 
down a connection from the Tomcat side if it is idle for to long.

If you don't want to reuse connections at all, there's also a setting (a 
JkOption in Apache).

> Is it possible that the Checkpoint Firewall in Between can be responsible for the connectivity

It can cut a connection that's idle for too long. Since you have 
cping/cpong active via connect_timeout and prepost_timeout, you should 
get a cping error message, if the connection was dropped by the firewall 
during idle times and mod_jk tries to use it again. The reply timeout in 
the error log indicates, that the backend isn't answering. Of course if 
it takes *very* long to answer, it might be that the firewall dropped 
the connection in between, but then the root cause would still be the 
long response time of the backend.

> Another point is the "not recovering" of the worker. Yes, you are right - in this situation
i have many reply_timeouts - but these happens in a period of time - for example 30 minutes
- but the worker is still dead even then when there are no more reply_timeouts. It remains
> It was necessary to restart it manually via jkstatus.

I assume you are using stickyness, so when a session started on a node, 
it will stay there. So when a worker is in error for a long time, all 
new sessions will start on other nodes. If the worker is ready for 
recovery, it needs a request, that doesn't carry a session to get probed 
with this request.

In jkstatus, the status of an error worker should switch to REC, when 
mod_jk decides that it could send a non-sticky request there (to probe) 
and to PRB, during the time this request is on the node, and finally 
either to OK or back to ERR depending on the result of the request.

You can log the number of errors (and accesses) that happened on the 
node in the httpd access log. If you think that the node simply stays in 
error for a long time, then the error count (and access count) should 
stay constant. I would expect, that they do not.

Have a look at how LogFormat in Apache httpd works, and then add some of 
those documented in



using the syntax %{JK_LB_LAST_STATE}n etc.

> Another point is the learning - i read the dics - the infos on the apache Website i dont't
find other ones - are there other ones ? - and they are not going in depth - if you read the
spec and watch the logs it is - for me - very hard to match the things. Also the many possibilities
that mod_jk has to prove if there is a connection to the Backend,... - i understand them but
check the reality in an error situation is very hard. Under matching i mean "Which Part of
the Communication sequence failed - why - and causes which error message".
> But i will try - and study also the mailing list..

It's hard for us too (sometimes).

> Thank you for your time - tomorrow we will have the new version and will see what happens.
> best
> ahmed



> -------- Original-Nachricht --------
>> Datum: Wed, 20 Feb 2008 15:56:42 +0100
>> Von: Rainer Jung <>
>> An: Tomcat Users List <>
>> Betreff: Re: mod_jk Problems - - worker went to error state and dont recover
>> wrote:
>>> See Thread at: Posted on
>> behalf of a User
>>> Hallo to all, After long unsuccessful research i hope someone can
>>> give me a hint to the following problems.
>>> Our Apache-mod_jk-Tomcat Infrastructur was running without Problems
>>> for about one year-than since two month mod_jk errors occurs.
>>> We upgraded the mod_jk Version, made improvements in the
>>> - the problems changed and get less but sometimes they
>>> appear further on.
>>> It seems that the mod_jk worker loose the connection to their
>>> Tomcat-Backendserver - there are messages in the mod_jk log Files which
>>> points in this direction. Normally this seems not to be a big problem -
>>> but under certain conditions (which ?) the worker goes to an error state
>>> and cannot recover itself- must be done manually.
>>> Problem 1: The Tomcats are reachable - unknown why the workers think the
>> server is dead ?
>>> Problem 2: I have no idea why the worker goes to an error state and
>> cannot recover.
>> 2 is a consequence of 1
>>> Problem3: I miss explanations of logged messages - i read the messages -
>> but cannot match them to the situation - when does a worker post this
>> messages
>> 1 is a consequence of these messages
>>> [Wed Feb 20 10:04:01.889 2008] [19237:3086010048] [info]
>> jk_handler::mod_jk.c (2270): Aborting connection for worker=ajp_ggi 
>>> [Wed Feb 20 10:04:39.799 2008] [19294:3086010048] [error]
>> ajp_get_reply::jk_ajp_common.c (1623): (INETP1011) Timeout with waiting reply from
>> tomcat. Tomcat is down, stopped or network problems (errno=110)
>>> [Wed Feb 20 10:04:39.799 2008] [19294:3086010048] [error]
>> ajp_service::jk_ajp_common.c (2034): (INETP1011) receiving reply from tomcat failed
>> out recovery in send loop attempt=0
>>> [Wed Feb 20 10:04:41.799 2008] [19294:3086010048] [error]
>> service::jk_lb_worker.c (1105): unrecoverable error 504, request failed. Tomcat failed
>> the middle of request, we can't recover to another instance.
>> The second line tells us, that your configured reply_timeout fired.
>> You set it to 120000 (2 minutes), so there are requests taking longer 
>> than 2 minutes on the backend, before the first response packet comes 
>> back from the backend.
>> With your configuration mod_jk then doesn't wait any longer on the reply 
>> *and puts the backend into error mode*.
>> Up until version 1.2.25, if you use a reply-timeout, you need to set it 
>> to a high number which justifies the resoning "if it takes that long, 
>> that something is wrong with the backend".
>> Reality shows: there is no such number. Often there are few requests 
>> that take unaccetably long on the backend *although* the backend is 
>> still working.
>> So in 1.2.25 we added max_reply_timeouts. With this set in addition to 
>> reply_timeout, mod_jk will abort waiting for a reply after 
>> reply_timeout, but allow some timeouts before actually deciding to put 
>> the backend into error.
>> Unfortunately the implementation of max_reply_timeouts in 1.2.25 was 
>> wrong, so you need to go to 1.2.26 to get it working right.
>> See:
>> Caution: this does *not* explain, why the backends are not automatically 
>> recovered after a minute of error condition. Maybe you have times, where 
>> you getr to many of those reply_timeouts (see log file), and although we 
>> recover after a minute the backend almost immediately goes back into 
>> error status.
>>> -> Which Timeout - how does mod_jk think Tomcat is down ? Where can i
>> found details to errno=110 ?...
>> reply_timeout, see above and also
>> errno: a standard unix feature. The numbers are platform dependent. I 
>> would assume in your case
>> ETIMEDOUT       110     /* Connection timed out */
>> so no wonder, that's exactly what we expect (and doesn't tell us the 
>> reason, i.e. what's wrong on the *backend* taking that long for a
>> response).
>>> -> receiving reply from tomcat failed with out recovery in send loop
>> attempt=0  - ? with out recovery in send loop - means?
>> That your configuration doesn't allow us to send the request to another 
>> backend. recovery_options 7 include: if mod_jk was able to send the 
>> request to a backend, do not try to send it to another backend in case 
>> of an error during the response handling. Even if you would allow 
>> sending to another backend, it would not help with *not* putting the 
>> worker into error state. More likely would be, that you would put all 
>> workers into error state, because all of them might run into the same 
>> timeout, one after the other.
>>> -> unrecoverable error 504 - details to this error ?
>> That's simply how we return the situation back to the client (browser).
>>> Ok - i turn the logging level to debug - the course of events get
>>> more
>>> clear - but also more questions appear - there are socket numbers -
>>> which sockets - what are these numbers e.g will be shutting down socket
>>> 35 for worker INETP1021 - The sockets are good for ? - how many are
>>> there/per worker ? can i configure them ?
>> Should not be the problem here. For apache httpd if you do *not* 
>> configure anything, we automatically choose the number of httpd threads 
>> as the maximum number of connections. No need to change anything here.
>>> => Generally -How can i solve such problems - i tried to look into
>>> the
>>> mod_jk code - searching for error codes, error messages - but cannot
>>> find some relevant informations, - i am studying the log Files - but
>>> don't find out what really happens.
>> Post to the list. Improve our dics.
>> The error message contains the word "timeout" and "reply" and you have a 
>> "reply_timeout".
>> Long running requests are a frequent problem. If you want to get rid of 
>> them, start by adding response times to your httpd and your tomcat 
>> access log format (%D). Then have a look, which URLs are producing long 
>> running requests, during what time of day are they happening etc. This 
>> might give you a clue about the reasons.
>> And if they are very frequent: do Java Thread Dumps of your backends and 
>> analyze them.
>>> So - maybe someone has an idea why the worker think that the
>>> corresponding Tomcat is dead, and why he will not recover by itself. !
>> Tomecat is dead: from the point of view of mod_jk it simply means: we 
>> didn't get an answer, when we expected one. Details depend on the 
>> additional log lines (could not connect, reply timeout etc.).
>>> And i am also searching for tips how i can help myself - and where to
>>> find something about the error codes, messages, mod_jk
>>> thanks for your attention
>>> Best
>>> ahmed musa (writing from vienna)
>> Regards,
>> Rainer
>>> Current Infrastructur
>>> We have 3 Apache Webserver (2.2.6) -based on CentOS release 4.3
>> /Kernelversion 2.6.9-34
>>> In front of the Webserver there are two (two Locations) HW-Loadbalancer
>> (but they have no role in this story)
>>> The Webservers are hosted at our ISP.
>>> The Webserver balance the requests via mod_jk (Version 1.2.25) for
>>> approx. 10 Webapps to 18 Backend-Tomcatserver (Bladeserver - because of
>>> underlying Application-Parts the OS is Windows 2003 Server - a long
>>> story not worth to explain :-) ). The Tomcatserver gain Data via
>>> Requests against DB2 Server/DB2-Databases on the Mainframe. The
>>> Tomcatserver are Inhouse -and were rebooted nightly because of automated
>>> Deployment processes.
>>> Between the Webserver and the Tomcatserver is a Checkpoint Firewall. 
>>> All webapps are deployed on all Tomcats - only mod_jk manages the
>>> requests to certain Tomcat- instances.
>>> (on one Bladeserver there are two identically Tomcat Instances
>>> running).
>>> Versions: Tomcat - 5.5.17_11, JDK 1.5.0_11-b03. The requests against
>>> the public Website(s) are normal short living requests - not many - The
>>> most Webapps (Portals) need a login, have a strong focus on business
>>> logic - so the instances are big (many MBs in RAM), the sessions are
>>> sticky and the session timeout is 20 minutes. But there are also less
>>> requests. To the User requests - Monitoring requests from our ISP are
>> added.
>>> The Problems appears at Servers/Portals which very less Userrequests.
>>> worker.list=ajp_bam,ajp_ggi,ajp_ad,ajp_svp,.......,jkstatus
>>> worker.template.type=ajp13
>>> worker.template.lbfactor=5
>>> worker.template.socket_keepalive=1
>>> worker.template.connect_timeout=7000
>>> worker.template.prepost_timeout=5000
>>> worker.template.reply_timeout=120000
>>> worker.template.retries=6
>>> worker.template.activation=Active
>>> worker.template.recovery_options=7
>>> worker.lbtemplate.type=lb
>>> worker.lbtemplate.max_reply_timeouts=6
>>> worker.lbtemplate.method=Session
>>> #Produktions Worker
>>> # AS-INETP101 - 106 - 6/6 GGI
>>> worker.INETP1011.port=65001
>>> worker.INETP1011.reference=worker.template
>>> ....many more of the same
>>> then
>>> worker.ajp_ad.reference=worker.lbtemplate
>>> worker.ajp_ad.balance_workers=INETP1032,INETP1062
>>> .... many more portals
>>> at least jkstatus
>>> The JKMount is very simple
>>> JkMount /* ajp_ad    --- for the other portals mostly the same
>>> The Portals are Virtual Hosts on the Apache.
>>> Tomcat - server.xml
>>> example
>>> <Connector port="65001" maxThreads="300" protocol="AJP/1.3" />
>>>     <Engine name="Catalina" jvmRoute="INETP5021" defaultHost="default">
>>> ......
>>> <Host name="" appBase="webapps" unpackWARs="true"
>>> autoDeploy="false" deployOnStartup="false" xmlValidation="false"
>>> xmlNamespaceAware="false">
>>>         <Alias></Alias>
>>>         <Alias></Alias>
>>>         ...
>>>         <Alias></Alias>
>>>         .....
>>>         <Valve className="org.apache.catalina.valves.AccessLogValve"
>>> directory="logs" prefix="swl_access_log." suffix=".txt" pattern="common"
>>> resolveHosts="false" />
>>>         <Valve
>>> className="at.allianz.tomcat.valve.RequestTimeValve"/>
>>>         <Valve
>>> className="at.allianz.tomcat.valve.WebcollaborationWorkaroundValve"/>
>>>         <Context path="" docBase="swl" />
>>>         <Context path="/monitor5" docBase="monitor" />
>>>         <Context path="/swl" docBase="swl" />
>>>       </Host>    

To start a new topic, e-mail:
To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message