Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (athena.apache.org: domain of rainer.jung@kippdata.de
 designates 195.227.30.149 as permitted sender)
Message-ID: <4B267789.8030204@kippdata.de>
Date: Mon, 14 Dec 2009 18:36:09 +0100
From: Rainer Jung <rainer.jung@kippdata.de>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de;
 rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Tomcat 6.0.16 + mod_jk 1.2.19 - request threads hanging up
References: <ce3aa0b60911180337o6469bdb4tcf3a90b44cea4591@mail.gmail.com>
 <99C8B2929B39C24493377AC7A121E21F967C706AA2@USEA-EXCH8.na.uis.unisys.com>
	 <ce3aa0b60911181253t3510f970u61eb5bf3ffa3b08b@mail.gmail.com>
 <26673416.post@talk.nabble.com>
 <ce3aa0b60912090300m255013fai4cdf1a36dfd4f682@mail.gmail.com>
	 <4B1F877D.9090502@pidster.com> <4B2264F9.3030002@kippdata.de>
 <ce3aa0b60912120426r6669e513ye22e464c2f5b13fa@mail.gmail.com>
	 <4B23BD5A.6090107@kippdata.de>
 <ce3aa0b60912140920g6f28c874la1695bad8cbe922b@mail.gmail.com>
In-Reply-To: <ce3aa0b60912140920g6f28c874la1695bad8cbe922b@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit

On 14.12.2009 18:20, Alessandro Bahgat wrote:
> On Sat, Dec 12, 2009 at 4:57 PM, Rainer Jung<rainer.jung@kippdata.de>  wrote:
>> On 12.12.2009 13:26, Alessandro Bahgat wrote:
>>>
>>> We actually found out a lot of "unrecoverable error 200, request
>>> failed" error messages in the mod_jk log (roughly around 1k per hour),
>>> so I'm starting to wonder if there's any issue with the firewalls and
>>> the network infrastructure. What would you think about that?
>>>
>>> [Thu Dec 03 16:58:52 2009][31539:42688] [info]
>>> service::jk_lb_worker.c (873): unrecoverable error 200, request
>>> failed. Client failed in the middle of request, we can't recover to
>>> another instance.
>>> [Thu Dec 03 16:58:52 2009][31539:42688] [info]  jk_handler::mod_jk.c
>>> (2056): Aborting connection for worker=applprod
>>
>> The above two lines belong together, the next lines are something different.
>> The pait [pid:tid} changed.
>>
>> The above lines are logged, whenever there was a problem sending back gthe
>> response from Apache to the client/browser. It may happen, if a user in the
>> meatime clicked on something else or pressed the reload button. If you get
>> it a lot, maybe your app is to slow, your users are to nervous, or indeed
>> there might be a network problem. Occasional occurrences are normal.
>
> Well, I saw that error on yesterday's logs about 25k times (roughly 1%
> of the total requests) and the website isn't particularly slow these
> days (it takes less than 1s to render entirely the homepage).
> When it happens, we have some connections stuck in the ESTABLISHED
> state with non-zero values in the Send-Q, and Java's CPU consumption
> increases significantly.

Check which thread is the CPU hog and try to identify what it is doing 
using Java thread dumps. Not sure here, what the original reason is, and 
what's just a consequence.

Either something takes a lot of CPU thus making your app slow and the 
client aborting the connection due to user interaction, or something is 
aborting the connection and your Tomcat/Webapp doesn't handle it right, 
leading to increased CPU. Thread dumps should tell us the solution.

In your case I'd say the increased CPU usage is an old bug in tcnative, 
that should have been fixed in more recent releases.

> We'll upgrade Apache and mod_jk and see if this anything changes.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org