tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Cherouvim ...@eworx.gr>
Subject mod_jk fails to forward request on high traffic
Date Tue, 08 Sep 2009 15:37:04 GMT
Hello

I have a website which during peak time (peak lasts around ~4 hours with 
14 pageviews/sec, 140 http requests/sec) starts to drop pageview requests.

My setup is:
- SUSE Linux Enterprise Server 10 (x86_64) with 8 cpus, 16GB ram
- 1 instance of apache 2.2.10 with mpm prefork
- mod_jk/1.2.28
- 1 instance of tomcat 6.0.20

the O/S, tomcat and mysql seem to be completely underutilized:
- cpu is at 2% to 4%
- tomcat page render time is very low, around 50-200ms
- iostat shows minimal disk utilization
- lots of free disk space

even when apache is not saturated, 1 out of ~6 requests (to be routed to 
tomcat) will never return (or will return after 5-10 seconds).
mod_status at that point shows me the following which means that it can 
handle more requests:

KK.K_____CKK___KC_KK.__C__K___K__._CCKKKC_K__KK_KW_KK____KW..KC_
KKK_KCK_KCC_.___W_CC__CC__KC.CK_K.CKK___KK_C_CKKK_KKKKCCKK.KKKK_
K___KW_CK_K.CK_.C____C.C..KCRW.._KC.._.K....K..K...K_.._..K..K.K
.KC._W.C...._.K_CC._.K__K..KC_._K...K.K._..._..._._......_CKK_..
_.CKK_K_.KK...............R.....................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................


All requests to static content (not served via tomcat) have no problem.

my apache config is:
KeepAlive On
MaxKeepAliveRequests 1024
KeepAliveTimeout 5

JkLogFile     logs/mod_jk.log
JkShmFile     logs/jk-runtime-status
JkLogLevel    info

JkWorkerProperty worker.list=tc1
JkWorkerProperty worker.tc1.type=ajp13
JkWorkerProperty worker.tc1.host=localhost
JkWorkerProperty worker.tc1.port=8010

<VirtualHost *:80>
    ServerName www.foobar.com
    DocumentRoot ...

    JkMount /* tc1
    JkUnMount /*.css tc1
    JkUnMount /*.js tc1
    ...
    JkUnMount /*.jpg tc1
</VirtualHost>

<IfModule mpm_prefork_module>
    StartServers         5
    MinSpareServers      5
    MaxSpareServers      10
    ServerLimit        1024
    MaxClients         1024
    MaxRequestsPerChild   100000
</IfModule>

my tomcat config is very plain:
<Service name="Catalina">
  <Connector port="8010" protocol="AJP/1.3" redirectPort="8443" 
address="127.0.0.1" />
  <Engine name="Catalina" defaultHost="www.foobar.com">
    <Host name="www.foobar.com" appBase="webapps/foobar" />
  </Engine>
</Service>


After I stop/start apache I start receiving these logs many times on 
mod_jk.log:
[5449:902382544] [info] ajp_process_callback::jk_ajp_common.c (1748): 
Writing to client aborted or client network problems
[5449:902382544] [info] ajp_service::jk_ajp_common.c (2407): (tc1) 
sending request to tomcat failed (unrecoverable), because of client 
write error (attempt=1)
[5449:902382544] [info] jk_handler::mod_jk.c (2469): Aborting connection 
for worker=tc1

Exactly 200 seconds after the restart (confirmed it 10 times) these 
start appearing as well:
[5761:902382544] [info] jk_open_socket::jk_connect.c (593): connect to 
127.0.0.1:8010 failed (errno=110)
[5761:902382544] [info] ajp_connect_to_endpoint::jk_ajp_common.c (922): 
Failed opening socket to (127.0.0.1:8010) (errno=110)
[5761:902382544] [error] ajp_send_request::jk_ajp_common.c (1467): (tc1) 
connecting to backend failed. Tomcat is probably not started or is 
listening on the wrong port (errno=110)

These logs may be unrelated to the slow or dropped responses from mod_jk.

On tomcat catalina.out I'm getting lots of:
org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2

Any ideas on where should I look at?
Will the mod_jk status worker give me any insights on this?
Is there any possibility that mod_jk is getting "blocked" at the O/S 
level (maybe from iptables)?

Also, I'm not sure if I can successfully reproduce the whole situation 
locally with JMeter since I don't have these many hosts to hit a staging 
server with.
All I'm left with is wait each day for the peak time and fiddle around 
with the settings of the production server.

thanks a lot
Ioannis

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message