tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier>
Subject Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Date Fri, 15 Mar 2013 08:41:50 GMT
David Kumar wrote:
> hey,
>> Yes.  Whether this is the java that runs tomcat is not immediately evident.
>> But it also seems that they concern port numbers like 7009, 9009 etc. which >must
be your 
>> AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat.
>> The first observation hat I would make is that this looks like a pretty >busy
>> Then, there are 2100+ lines in that list. Let's assume first that they are >all
>> connections (which probable they are not, but for simplicity), so that >there
are actually 
>> 2 lines for the same connection, like these :
>> tcp        0      0          >FIN_WAIT2

>> 20766/apache2
>> . ..
>> tcp6       1      0         >CLOSE_WAIT
>> ..
>> So let's say that we have in total about 1000 connections active in some >state.
>> Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%.
>> That does not really sound "pathological" to me, considering that >CLOSE_WAIT
is a normal 
>> state through which any TCP connection goes at some point.
>> If you look again at the above connection, and if I am not confusing my >clients
>> servers again, it looks here like :
>> - Apache http (PID 20766), as a client, has established a connection to >port
7009 of the 
>> server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk >module
>> Apache, to the AJP Connector in Tomcat.
>> - A while later, Apache closes the connection. It sends a "FIN" packet to >Tomcat,
>> waits for Tomcat to acknowledge this FIN packet.
>> - when it receives the FIN packet from Apache, Tomcat responds with an ACK, >and
then goes 
>> to the CLOSE_WAIT state.
>> - when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2.
>>   (and that is the current state of the above connection)
>> - now Tomcat is supposed to send a FIN to Apache, and receive an ACK in >return.
>> - if Apache receives the FIN from Tomcat, it sends back an ACK, and goes >into
>> TIME_WAIT (where it doesn't expect anything anymore), and then after a >short
moment, it 
>> discards that connection.
>> If there are connections that remain in the CLOSE_WAIT state for very long, >it
means that 
>> something in the last 2 steps above is not working.
>> Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of >them
*to* an 
>> AJP port of Tomcat.  This seems to suggest that the Apache side is working >as
it should.
>> But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would >suggest
that on 
>> the Tomcat side, that connection is never properly close()'d.
>> But this is the request/response connection, so I doubt that there would be >a
bug there, 
>> otherwise we'd have problems reports filling this list every day.
>> Might there be that there is somewhere a discrepancy between the keep-alive >settings,

>> between Apache and Tomcat ?
>> or maybe your server is just not strong enough for the load you are putting >on
it ?
> According to vmtools the server is boring most times.. 

What do you mean, "boring" ? (like German "langweilig" ?).
If you have 2000+ connections there in one state or another, that should mean that the 
system is usually pretty busy, I would think.  On my production machines (vSphere VM's, 
Linux, Apache+Tomcat, +database), some of them pretty busy, I never see more than 200-300

connections with netstat. That is all states together, with "netstat -t -pan | wc -l").
Granted, I am not running Yahoo or Facebook, but still, these are public Internet servers,

fairly sollicited.

> I almost totally agree, but I'm not sure if it is a AJP fault. Because most time everything
is alright, so all of theses steps are done. Just sometimes after a arbitrarily amount of
time ajp / tomcat runs crazy. And in my sample there is the sourceport 9009 / java.  It looks
for me that apache didn't received the ACK from tomcat after CLOSE_WAIT or it just don't care
because apache thought this connections is already closed?  Let's assume that is the case..
Isn't there a option to tell tomcat to wait max. time for a response and after that time kill
the connection? As far as I understand connectionTimeout is just for the initial request,
so after Apache did answer in first correctly connectionTimeout is valueless?
Let's look at the various options in the Connector :

First, the "-1" in some options means "unlimited" or "forever", it doesn't mean "disabled".
Second, some parameters say that if they are not specified, they default to the value of 
another parameter, so you should check really what this other parameter is set to.

> Attached you can find our configuration files, maybe you can find a / some misconfigurations?

I'll have a look.

> I would agree that we have maybe a kind of misconfigured timeout problem...
> thanks
> Mit freundlichen Grüßen
> David Kumar  
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv 
> Am Weiher 14
> D-56766 Ulmen
> ------------------------------------------------------------------------
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message