tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: jk-to-tomcat multiple retries
Date Sat, 02 May 2009 11:34:55 GMT
On 01.05.2009 18:19, Dmitry Beransky wrote:
> We have the strangest problem started happening to us a few weeks ago
> (after several years of running pretty much the same configuration).
> 
> 1. The problem is only happening in the production environment.  We
> cannot reproduce it on staging, which as far as we can tell is
> configured identically to production.
> 2. The problem seems to be tied to the traffic volume and possible
> pattern (Hence probably why we cannot reproduce it on staging).
> 3. Our configuration:  W2K server running IIS 6 and JK 1.2.27, Tomcat
> v. 5.5.12 running on the same box.
> 
> The problem is as follows: some requests would result in multiple
> copies of the first buffer-full of expected data, ended either by a
> 503 error page data, a 502 error page data, or the rest of the proper
> page.  The number of copies is directly related to the JK's number of
> retries setting.  With the number of retries initially being set to
> 10, the maximum number of repeated copies in the response was 20.
> When we set the number of retries to 2, invalid replies contain only a
> single buffer-full of the page's proper data followed by an error page
> data.  On the Tomcat side, these multiple copies show up as multiple
> request entries in the access log, while there is only one
> corresponding request entry in the IIS log.
> 
> After a fresh restart of Tomcat, it takes a little while for this
> problem to start manifesting itself.  With time, it is starting to
> affect increasingly more requests until finally Tomcat gets entirely
> locked up.  At that point Tomcat needs to be restarted.... lather,
> rinse, repeat...
> 
> Here's what a sample of error messages in JK's log looks like:
> 
> [2760:2476] [error] jk_isapi_plugin.c (1199): WriteClient failed with
> 10053 (0x00002745)
> [2760:4020] [error] jk_ajp_common.c (1726): Chunk length too large.
> Length of AJP message is 8188, chunk length is 8192.
> [2760:4020] [error] jk_ajp_common.c (2426): (default_1) connecting to
> tomcat failed.
> [2760:1104] [error] jk_ajp_common.c (1726): Chunk length too large.
> Length of AJP message is 8188, chunk length is 8192.
> [2760:1104] [error] jk_ajp_common.c (2426): (default_1) connecting to
> tomcat failed.
> [2760:1104] [error] jk_ajp_common.c (1726): Chunk length too large.
> Length of AJP message is 8188, chunk length is 8192.
> [2760:1104] [error] jk_ajp_common.c (2426): (default_1) connecting to
> tomcat failed.
> [2760:1104] [error] jk_lb_worker.c (1432): All tomcat instances
> failed, no more workers left
> [2760:1104] [error] jk_isapi_plugin.c (2199): service() failed with
> http error 503
> [2760:3876] [error] jk_ajp_common.c (1726): Chunk length too large.
> Length of AJP message is 8188, chunk length is 8192.
> [2760:3876] [error] jk_ajp_common.c (2426): (default_1) connecting to
> tomcat failed.
> [2760:3232] [error] jk_ajp_common.c (1726): Chunk length too large.
> Length of AJP message is 8188, chunk length is 8192.
> [2760:3232] [error] jk_ajp_common.c (2426): (default_1) connecting to
> tomcat failed.
> [2760:3232] [error] jk_ajp_common.c (1726): Chunk length too large.
> Length of AJP message is 8188, chunk length is 8192.
> [2760:3232] [error] jk_ajp_common.c (2426): (default_1) connecting to
> tomcat failed.
> [2760:3232] [error] jk_lb_worker.c (1432): All tomcat instances
> failed, no more workers left
> [2760:3232] [error] jk_isapi_plugin.c (2199): service() failed with
> http error 503
> 
> The Chunk length messages have been in our logs forever.  Yesterday I
> temporarily changed JK & Tomcat configuration matching the packet
> sizes.  The chunk errors went away, but the problem seemed to persist,
> so I put everything back the way it was.

The chunk length message seems pretty weird. Looks like a protocol
corruption. Those indicate, that you should really try a TC update.
Concerining your restriction "can't update before any other options are
exhausted": there will never be any other options exhausted. But after
some options are taken, the rest get more and more expensive, risky and
with a low chance of success.

> To me this look likes some weird error condition in Tomcat has hit an
> obscure bug in JK whereby it doesn't clear the response buffer between
> retries.  Has anyone encountered this issue before or is just willing
> to land a helping hand in troubleshooting?

Not encountered this before, and I think noone reported a similar
observation. Concerning "retries": Could you provide your full
configuration (e.g. retries for an ajp13 worker is something very
different from retries of a load balancer worker).

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message