Mailing-List: contact tomcat-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <tomcat-user@jakarta.apache.org>
Received-SPF: pass (hermes.apache.org: local policy)
Message-ID: <417DF229.2060202@bb-sw.de>
Date: Tue, 26 Oct 2004 08:43:53 +0200
From: Christoph Fischer <cf@bb-sw.de>
User-Agent: Mozilla Thunderbird 0.8 (Windows/20040913)
MIME-Version: 1.0
To: Tomcat Users List <tomcat-user@jakarta.apache.org>
Subject: Re: Problems wth Apache, mod_jk2 and Tomcat
References: <417D4779.4060909@worldlingo.com> <417D4E4B.2030703@cornell.edu>
 <417D787A.5010208@worldlingo.com> <417D97AB.6000306@cornell.edu>
In-Reply-To: <417D97AB.6000306@cornell.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


Hi,
this seems also to be a problem in mod_jk , tomcat4.1.10, apache 1.3 on 
a linux server however it does build up slowly over the day.
The problem seems to be that the connection via mod_jk (Port 8009) does 
not close, so the java/tomcat processes
will not quit after responding to the request (or the other way 
around?). This problem occurs only in
heavy load situations. After that I see many open socket connections an 
some tomcat processes that will not quit.

Snapshot of the logfiles:

tomcat / catalina.out:
Failed to authentice as null
Error number: 50

tomcat / catalna_log:
2004-10-22 08:52:29 Ajp13Processor[8009][358] Starting background thread
2004-10-22 08:52:34 Ajp13Processor[8009][359] Starting background thread
2004-10-22 08:52:34 Ajp13Connector[8009] No processor available, 
rejecting this connection
2004-10-22 08:52:34 Ajp13Connector[8009] No processor available, 
rejecting this connection

apache / mod_jk
[Fri Oct 22 08:52:34 2004]  [jk_ajp_common.c (738)]: ERROR: can't 
receive the response message from tomcat, network problems or tomcat is 
down. err=-104
[Fri Oct 22 08:52:34 2004]  [jk_ajp_common.c (1137)]: Error reading 
reply from tomcat. Tomcat is down or network problems.

Is this a mod_jk problem not closing the connections or a tomcat problem 
not telling mod_jk that it's finished or a system problem not
releasing the sockets ?

Thanks for any ideas...
Chris

David Smith wrote:

> Hmmm..  My assumption to your email was their was some kind of 
> possible probing of your Apache server and/or a misbehaving client.  
> Do you run snort on the box hosting the Apache httpd service?  It's an 
> intrusion detection tool designed to log suspicious activity.  That 
> could be something to look at.  Otherwise I can't think of a reason in 
> the world for httpd to spontaneously go full out.  It's not something 
> I've ever seen.
>
> --David
>
> Lars George wrote:
>
>> David,
>>
>> This proves more difficult, since the requests look like standard 
>> requests that work at other times. Moreover the POST data is no 
>> logged anyway so I cannot check if it was a value that was sent in by 
>> chance.
>>
>> Is there anything else I can check to see what is going on? I was 
>> more thinking along the lines of using some low-level tools to see if 
>> everything is OK, for example critical resources related to Apache or 
>> Tomcat (note, they are on separate machines). For example I tried 
>> using "netstat -p" on either side to see what the connections are 
>> saying. I als did a thread dump of the Tomcat JVM to see what the 
>> Coyote connectors and all the other threads are up to. But There is 
>> nothing conclusive so far. Just out of the blue the apache runs full, 
>> and there seems no relation to when during the day or how high the 
>> load etc.
>>
>> Thanks,
>> Lars
>>
>>
>> David Smith wrote:
>>
>>> I would start with the apache logs and find out what kind of 
>>> requests are logged in the access just before the event.  That 
>>> should get you going in the right direction.
>>>
>>> --David
>>>
>>> Lars George wrote:
>>>
>>>> Hi,
>>>>
>>>> We have an odd problem we cannot solve. Maybe someone else has come 
>>>> across this too.
>>>>
>>>> We use Apache 2.0.52 with the Tomcat 5.0.25 and its included 
>>>> mod_jk2 with Coyote/JK2 AJP 1.3 connector.
>>>>
>>>> Usually Apache uses 230 slots out of 1000 it has set as the 
>>>> maximum. This can be seen from the server-status page, it equals to 
>>>> say 32 requests per second.
>>>>
>>>> Then all of sudden the Apache runs full and exceeds all empty 
>>>> slots, ie. goes up to 1000 requests currently being processed and 
>>>> accepts no further connections. All but a handful of these slots 
>>>> are in state "W". Looking at the URI's the requests are both static 
>>>> content (like images) served directly by Apache, but of course many 
>>>> requests are to the dynamic content created by the app server 
>>>> (Tomcat).
>>>>
>>>> I do not think this is a load issue per se, ie. where we have 
>>>> someone hammering us with requests, since looking at the 
>>>> connections and URL's they are all so random but valid at the same 
>>>> time that I do not think this is it.
>>>>
>>>> What puzzles me more is that they are all in the state "W". which 
>>>> means "sending reply". The network seems to be OK at that point of 
>>>> time too since I can connect to the machines no problem.
>>>>
>>>> Only if I restart Apache by ultimately killing (killall -9 httpd) 
>>>> all processes I can revive the site. After the Apache restart 
>>>> everything goes back to normal right away. This seems to mean 
>>>> something too, at least that it seems to be getting stuck somehow 
>>>> but whatever caused it does not seem to exist anymore. Like 
>>>> something short, for example a specific request, caused this.
>>>>
>>>> What could I check from here? I cannot take the Apache out since we 
>>>> need  it for rewriting and session handling.
>>>>
>>>> Any help or pointers are greatly appreciated.
>>>>
>>>> Thanks,
>>>> Lars
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org