tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Beransky <dmitry.ma...@gmail.com>
Subject jk-to-tomcat multiple retries
Date Fri, 01 May 2009 16:19:54 GMT
Hi,

We have the strangest problem started happening to us a few weeks ago
(after several years of running pretty much the same configuration).

1. The problem is only happening in the production environment.  We
cannot reproduce it on staging, which as far as we can tell is
configured identically to production.
2. The problem seems to be tied to the traffic volume and possible
pattern (Hence probably why we cannot reproduce it on staging).
3. Our configuration:  W2K server running IIS 6 and JK 1.2.27, Tomcat
v. 5.5.12 running on the same box.

The problem is as follows: some requests would result in multiple
copies of the first buffer-full of expected data, ended either by a
503 error page data, a 502 error page data, or the rest of the proper
page.  The number of copies is directly related to the JK's number of
retries setting.  With the number of retries initially being set to
10, the maximum number of repeated copies in the response was 20.
When we set the number of retries to 2, invalid replies contain only a
single buffer-full of the page's proper data followed by an error page
data.  On the Tomcat side, these multiple copies show up as multiple
request entries in the access log, while there is only one
corresponding request entry in the IIS log.

After a fresh restart of Tomcat, it takes a little while for this
problem to start manifesting itself.  With time, it is starting to
affect increasingly more requests until finally Tomcat gets entirely
locked up.  At that point Tomcat needs to be restarted.... lather,
rinse, repeat...

Here's what a sample of error messages in JK's log looks like:

[2760:2476] [error] jk_isapi_plugin.c (1199): WriteClient failed with
10053 (0x00002745)
[2760:4020] [error] jk_ajp_common.c (1726): Chunk length too large.
Length of AJP message is 8188, chunk length is 8192.
[2760:4020] [error] jk_ajp_common.c (2426): (default_1) connecting to
tomcat failed.
[2760:1104] [error] jk_ajp_common.c (1726): Chunk length too large.
Length of AJP message is 8188, chunk length is 8192.
[2760:1104] [error] jk_ajp_common.c (2426): (default_1) connecting to
tomcat failed.
[2760:1104] [error] jk_ajp_common.c (1726): Chunk length too large.
Length of AJP message is 8188, chunk length is 8192.
[2760:1104] [error] jk_ajp_common.c (2426): (default_1) connecting to
tomcat failed.
[2760:1104] [error] jk_lb_worker.c (1432): All tomcat instances
failed, no more workers left
[2760:1104] [error] jk_isapi_plugin.c (2199): service() failed with
http error 503
[2760:3876] [error] jk_ajp_common.c (1726): Chunk length too large.
Length of AJP message is 8188, chunk length is 8192.
[2760:3876] [error] jk_ajp_common.c (2426): (default_1) connecting to
tomcat failed.
[2760:3232] [error] jk_ajp_common.c (1726): Chunk length too large.
Length of AJP message is 8188, chunk length is 8192.
[2760:3232] [error] jk_ajp_common.c (2426): (default_1) connecting to
tomcat failed.
[2760:3232] [error] jk_ajp_common.c (1726): Chunk length too large.
Length of AJP message is 8188, chunk length is 8192.
[2760:3232] [error] jk_ajp_common.c (2426): (default_1) connecting to
tomcat failed.
[2760:3232] [error] jk_lb_worker.c (1432): All tomcat instances
failed, no more workers left
[2760:3232] [error] jk_isapi_plugin.c (2199): service() failed with
http error 503

The Chunk length messages have been in our logs forever.  Yesterday I
temporarily changed JK & Tomcat configuration matching the packet
sizes.  The chunk errors went away, but the problem seemed to persist,
so I put everything back the way it was.

To me this look likes some weird error condition in Tomcat has hit an
obscure bug in JK whereby it doesn't clear the response buffer between
retries.  Has anyone encountered this issue before or is just willing
to land a helping hand in troubleshooting?


Thanks
Dmitry

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message