tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: mod_jk ERR/FRC
Date Mon, 03 Mar 2008 22:16:05 GMT
Hi Tim,

tim.fulcher@bt.com schrieb:
> Setup: TC 4.1.18, IIS5, mod_jk 1.2.25, JDK 1.3.1, Windows 2000
> 1 LB pointing to 1 worker TC, same server as IIS. Because of 3rd party
> license issues, we're stuck with using 1 TC only.
> 
> Usually the tomcat runs with ~ 280 threads, but soon after a bounce its
> getting 400+ threads, and pretty soon I get errors such as this -
> 
> [Error] ThreadPool- - Caught exception executing
> org.apache.jk.common.SocketAcceptor... <java.lang.OutOfMemoryError
> unable to create new native thread>

OK, usually when you run OutOfMemory during thread creation, this will 
also mean, that your Tomcat doesn't accept any new connections any more.

With the old style connector, whenever a new connection comes in, Tomcat 
accepts the connection on the thread, that waits for new connections and 
lets this thread work on the requests coming in via this connection.

It then looks for another thread to again accept new connections. If the 
pool is empty and the maximum number of threads is not reached, it will 
try to create a new thread. If there is not enough memory for this, 
thread creation will fail and from that on, there will be no more thread 
accepting any new connection. This effectvely means, that Tomcat will 
still answer the open requests, but no more new requests are handeld - 
until Tomcat restart.

That's the reson, why you should always limit the number of threads in 
the pool. It's better to not accept new connections, if your threads 
reahed the defined limit, than to create to many threads and by this 
break the service unrecoverable.

How many threads will work? It depends on the OS. On Windows something 
between 400 and 500, on Linux it depends on 32/64 Bits and the thread 
library. On Solaris Sparc there is no such trouble. It's mostly a 
question of memory adressing for thread stacks.

I would test, if 400 work, and then limit the pool to 400. How do you 
test? Simply configure the pool to 400 minimum and maximum and see, if 
you can start Tomcat and if all threads get created. Then you know, that 
your platform can create that many threads inside the JVM.

So far this is totally independent of mod_jk/isapi plugin.

> Shortly after, mod_jk declares the worker to be in an Error state, and
> all my site serves is the 'Service Temporarily Unavailable' page. The
> isapi_redirect.log is dumping out hundreds of messages including 'all
> tomcat instances are busy or in error state'. This is like a couple of
> minutes after TC is restarted !

The cat, that you get many such errors, once your Tomcat isn't accepting 
any more connections seems normal. The fact, that *all* tomcats are busy 
seems normal too, since you only have one.

Why will mod_jk put your Tomcat into error state. although it still has 
400 working connections (but cannot open a new one): because that's the 
way it's implemented. If we try to open a new connection and we get an 
error (e.g. a network timeout), we declare the backend as broken (error 
state).

> worker.properties has 
> worker.wkr.connection_pool_size=200
> worker.wkr.connection_pool_timeout=300

Hmmm. This would mean, that the web server only opens up max. 200 
connections to Tomcat. If we assume, that there is only one web server 
in front of Tomcat and no traffic is going to Tomcat from somewhere else 
(other connectors etc.), we should not see more than 200 and a few 
threads inside Tomcat. This contradicts your observations.

It's possible, that old connections are not closed correctly on the 
Tomcat side, if a firewall between web server and Tomcat drops idle 
connections. Usually one uses a connectionTimeout on the Tomcat side 
together with the connection pool timeout on the mod_jk side.

See:

http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html

> - When mod_jk shows a state of ERR/FRC, what exactly does the forced
> recovery attempt to do ?

If a balancer has to put all members into error state, it could either 
always return an error, or it could say: I've got no good backend, so 
lets try to use the bad ones anyhow. Force recovery means: no backend is 
OK, then use a bad one. In your case (one backend) it means, that 
although the backend goes into error, we still send the requests there.

> - Our webapp in TC works hard on startup to cache frequent content into
> memory. That means bouncing the Tomcat is painful for us during the day.

Fix an upper thread limit. If your app behaves well, then you shouldn't 
have to restart. Of course if the problem is bad performance and your 
ressources are exhausted, that problm will only end, if the load 
decreases, or the performance gets better. But usually the container 
will then proceed as normal, without restart.

In the case described abive, when the container could not start a new 
thread for socket acept, it will be broken and a restart is needed.

> I wonder anectdotally if the FRC messages from mod_jk are causing it
> hassle ? Moreover in the first ~15minutes of the webapp, response times
> will be longer than usual. Is this causing mod_jk to give up? If so
> whats the best tuning approach?

It will only give up on long response times when a reply_timeout is set.

> - Is there a simple way in mod_jk to throttle traffic incident on tomcat
> ?

No. There are other modules, but most of them throttle traffic per 
client IP. Search for httpd modules with "throttle" or "bandwidth". You 
could code a servlet filter though. Throttling inside Tomat is less 
stable than in front of it, but you can have some easy logic there, whom 
you want to send the throttle page (error page), e.g. to many users 
logged on (or to many sessions) and the request doesn't belong to a 
session. etc. Ooops, sorry, you are on IIS, forgot about that.

> - Lastly, can the 'Service Temporarily Unavailable' page be customised ?
> I'm presuming that its being served by mod_jk internals rather than IIS
> ?

Anything inside IIS? It would be ErrorPage with httpd.

> 
> regards
> 
> Tim

Regards,

Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message