tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: Error 503 on shutting down instance of failover cluster
Date Wed, 01 Oct 2008 15:03:14 GMT
MÃ¥rten Svantesson wrote:
> Hi,
> 
> This is actually a question regarding the combination of JBoss 4.0.2RC1 with Tomcat 5.0.28
as a component, and Apache 2.2.3 with mod_proxy_ajp. I hope
> this is the correct forum.

mod_proxy_ajp got several patches during the last couple of httpd 
releases. Although very likely none of them is related to your problem, 
you should keep your httpd recent in order to get those mod_proxy_ajp 
releases. At the moment version 2.2.9 is the latest and 2.2.10 is very 
close to get released (in about 1-2 weeks).

> To get failover we have two machines, both running Apache and JBoss. mod_proxy_ajp is
configured on both machines to do load balancing between the two
> JBoss instances. We also use sticky sessions.
> 
> The problem is that if you shut down JBoss on one machine (typically to upgrade our application)
and the JBoss currently is "busy" Apache may give you
> errors: Status 503 Service temporarily unavailable.

So the first question would be: what do you expect? Do you expect the 
request to fail over to the other node and return some application response?

If so: where does the 503 come from, is it a JBOss answer, or an answer 
from mod_proxy_ajp? If you can reproduce it on a test system, you can 
set your Apache LogLevel to debug and check the Apache error log to find 
out.

The shutdown case might be more3 complicated than the failure case. In 
the shutdown case the backend for some time still responds, possibly 
with error pages. I'm not sure, if you can configure with mod_proxy_ajp, 
that certain error codes should be handled like a failed backend. With 
mod_jk you could.

I would suggest if you do a known maintenance and shut down your 
backend, first stop the backend in mod_proxy_balancer, so that it knows 
it should send requests to this node any longer.

> I can see the following lines in the log:
> 
> 2008-09-10 18:40:33,664 DEBUG [tomcat.localhost.ROOT.Context] Stopping
> 2008-09-10 18:40:33,665 DEBUG [tomcat.localhost.ROOT.Context] Stopping filters
> ....
> A bunch of filters are shut down.
> ....
> 2008-09-10 18:40:33,679 DEBUG [tomcat.localhost.ROOT.Context] Processing standard container
shutdown
> 2008-09-10 18:40:33,685 INFO  [org.apache.catalina.core.StandardWrapper] Waiting for
14 instance(s) to be deallocated

The time di8fference between the above log lines and those here is only 
21 milliseconds. So no problem like "Jboss needs too long to shutdown 
and during that time mod_proxy_ajp returns 503" up until that point.

> It seems like new requests that happen to be given to this Tomcat (by proxy_ajp) are
not getting any responses until somewhere around when this turns
> up in the log:
> 
> 2008-07-08 14:27:39,622 DEBUG [org.jboss.web.tomcat.tc5.Tomcat5]  Destroyed jboss.web:service=WebServer

That entry is from a different day, so we cannot see, how long it took 
:) If you think, JBoss takes to long to shut down, create a couple of 
thread dumps during that time and look at them, which threads are still 
doing something and prevent JBoss from shutting down.

> It seems like the Tomcat threads don't disconnect (from proxy_ajp) until they are destroyed,
even though no incoming requests are handled. The gap in
> time between stopping and destroying Tomcat seems to be due to dependencies on components
(MBeans) that are busy.

How do you know, and what do you mean by "busy mbeans"?

> I can't see any reason for the idle connections to be open, so maybe it is possible to
close those on stopping one of the MBeans of our application.
> What do you think?

Usually the idle connections do not prevent JBoss or Tomcat from 
shutting down quickly, as long as they are really idle.

If you want to reduce the number of idle connections, you need to set 
the appropriate timeouts with mod_proxy and the backend connector.

> We used mod_jk with Apache 1.3 before and the problem got a slightly different symptom
there, instead of getting an error message the response from
> Apache was empty.

Never saw that, but most of the above applies here as well.

Regards,

Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message