tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: Delays in mod_jk
Date Sat, 03 Nov 2007 22:37:43 GMT
Hi Larry,

your analysis is correct, but the results nevertheless a little obscure.

Yes, mod_jk has no internal timing of management threads. We will move
mod_jk to the APR libraries when starting JK3, but at the moment we rely
completely on the web server plugin API. Apache has no timing API, so we
need to pack our maintenance on top of request handling.

Nevertheless, we don't expect the shutdown process to take that long. In
fact, there is a timeout of 2 seconds only used for select in a loop
that tries to read pending data from a connection we want to shut down.
Usually this draining should not really happen.

When I test the code on Solaris, I see no waiting in the select for a
connection, that is idle. The select returns betwen 0 and 2 milliseconds
after calling it (at least that's what I get from logging).

So now the questions are:

- what is your OS?
- are there any network components between httpd and jboss, which could
influence connection shutdown negatively (firewall etc.)?
- was jboss very busy during this test?

If you can easily reproduce the behaviour, can you sniff a connection?

How do I test?

I use a simple JSP sleep.jsp:

====
<%@page session="false"%>
Sleeping ...
<%
Thread.sleep(100);
%>
Done.
=====

For testing I configure connection_pool_timeout very low, e.g. 10
seconds and connection_pool_minsize to 1.

Then I send a couple of concurrent requests to the JSP to open a couple
of connections. For this I use the Apache httpd tool ab contained in the
bin directory

bin/ab -c 5 -n 100 http://myhost:myport/sleep.jsp

You can check via "netstat -an" if a couple of backend connections are
ESTABLISHED.

Now wait a minute and call the JSP once, e.g. with a browser.
mod_jk should then close all backend connections used by the httpd
process that handles this request (apart from 1 remaining connection).

If this works for you too, you can sniff the backend traffic after the
ab run and before the final request.

Regards,

Rainer

Larry Reisler schrieb:
> We are running JBOSS 4.21GA(Tomcat 6.0.10) with mod_jk 1.25 on Apache
> 2.2.x, and have run into something of a performance issue.
> 
> We have the system setup so that threads on mod_jk and tomcat timeout
> after 10 minutes (an heuristic which has been discussed in some of
> the documentation) using the properties on the mod_jk side: 
> worker.node1.connection_pool_timeout=600,and on the Tomcat side
> connectionTimeout="600000".
> 
> We have noticed that some transactions to the apache server were
> running slow.  After looking at the logs, the mod_jk logs indicated
> that the transaction executed relatively quickly (eg. < .03 seconds)
> whereas the apache access logs indicated that the transaction took
> something more like 2.040 seconds.  After enabling trace logs on
> mod_jk, it became clear that sometimes before executing a
> transaction, mod_jk would call the maintain function, which in turn
> would close sockets that had exceeded the time limit of 10 minutes
> using jk_shutdown_socket.   As per the code message in
> jk_shutdown_socket, the socket would take two seconds to shut down,
> causing a delay (sometimes it would shut down two or three sockets,
> resulting in a delay of 2,4, or 6 seconds).  The time spent doing the
> maintenance was before the real work of sending the data to Tomcat,
> so the mod_jk.log did not reflect that in its timing.  Nevertheless,
> the transactions are being delayed.
> 
> I found that the number of delays was directly related to the
> connection_pool_timeout: a higher timeout meant fewer delays – this
> is what I would have expected given my understanding of the problem.
> 
> I tried setting worker.maintain on the mod_jk side to 10 seconds, but
> the problem still occurs.
> 
> A couple of questions come out of this: 1.  Should the thread
> processing the request be running maintain at all?  My understanding
> is that there is a separate maintenance thread that is running by
> default every 60 seconds.  Can’t that thread perform the maintenance
> exclusively? 2.  If the processing thread does have to run the
> maintain function, then maybe it could simply place the expired
> sockets on a queue for later processing by the maintain thread, or
> perhaps send the FIN packet and then place them on a queue. 3.  Is
> there some reason that Tomcat would not be sending the FIN packet to
> mod_jk immediately?
> 
> Thanks Larry Reisler

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Mime
View raw message