tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From netg...@ipudev.com
Subject Load Balanced Tomcat freezing under high load
Date Wed, 26 Feb 2003 20:35:29 GMT

  Hi all.. we're running Apache 2.0.44, mod_jk2 2.0.2 and Tomcat 4.0.6 (all
compiled from source) load-balanced across 4 machines. Occasionally
(usually at peak times) one of the systems (usually the first one listed
in the workers2.properties) will freeze up and the CPU will go through the
roof. 'top' shows that its one particular java process consuming 99% of
the CPU. (I think it's one of the first ones spawned when tomcat is
started). At this point the Apache hits it's MaxClients (512) and the site
goes down, despite the other Tomcat workers being alive and well. I'm
pretty sure it's not an Apache problem as I just upgraded from
1.3.27/mod_jk-1.2 last night and the problem has happened again twice
today.

  I'm getting a lot of these in the Apache error log:
--------------- error_log
[Wed Feb 26 01:22:47 2003] [error] jk2_init() Can't find child 26949 in scoreboard
[Wed Feb 26 01:22:47 2003] [error] mod_jk child workerEnv in error state 3
[Wed Feb 26 01:22:47 2003] [error] mod_jk child init 1 -2
[Wed Feb 26 01:22:49 2003] [error] mod_jk child workerEnv in error state 3
[Wed Feb 26 01:22:50 2003] [error] mod_jk child workerEnv in error state 3
--------------- etc..etc
  and a few of these when the site got busier:
---------------
[Wed Feb 26 10:30:04 2003] [error] Error ajp_process_callback - write failed
[Wed Feb 26 10:30:04 2003] [error] ajp13.service() ajpGetReply recoverable error 3
[Wed Feb 26 10:30:04 2003] [error] Error ajp_process_callback - write failed
[Wed Feb 26 10:30:04 2003] [error] ajp13.service() ajpGetReply recoverable error 3
[Wed Feb 26 10:30:04 2003] [error] ajp13.service() Error  forwarding ajp13:jvm1:8009 1 0
[Wed Feb 26 10:30:04 2003] [error] mod_jk.handler() Error connecting to tomcat 3
[Wed Feb 26 10:30:13 2003] [error] mod_jk child init 1 0
---------- etc etc..
  I'm also seeing a lot of these in the catalina_log on the Tomcat
workers:
----------
2003-02-26 09:50:51 Ajp13Processor[8009][133] process: invoke
java.net.SocketException: Socket closed
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:126)
        at org.apache.ajp.Ajp13.send(Ajp13.java:525)
        at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
        at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
        at
org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
        at
org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
        at
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
        at java.lang.Thread.run(Thread.java:536)
---------

  Beyond that, the logs don't tell me much. Anyway, sorry for the huge
length of this e-mail; if anyone has any suggestions it would be a huge
help.

  Thanks!
  -Mike
   netguru at ipudev dot com


-------- workers2.properties
[shm]
file=/usr/local/apache/logs/shm.file
size=4194304

[channel.socket:jvm2:8009]
port=8009
host=192.168.0.31
group=lb
lb_factor=1
disabled=0
route=jvm2

[channel.socket:jvm1:8009]
port=8009
host=192.168.0.30
group=lb
lb_factor=1
disabled=0
route=jvm1

[channel.socket:jvm3:8009]
port=8009
host=192.168.0.32
group=lb
lb_factor=1
disabled=0
route=jvm3

[channel.socket:jvm4:8009]
port=8009
host=192.168.0.33
group=lb
lb_factor=1
disabled=0
route=jvm4

[various uri mappings to lb worker...]
---------
--------- server.xml (same on all Tomcat servers)
<Server port="8005" shutdown="SHUTDOWN" debug="0">
  <Service name="Tomcat-Standalone">
    <Connector className="org.apache.ajp.tomcat4.Ajp13Connector"
               port="8009" minProcessors="5" maxProcessors="500"
               acceptCount="100" debug="0"/>
    <!-- jvmRoute is configured appropriately for each system -->
    <Engine jvmRoute="jvm1" name="Standalone" defaultHost="localhost" debug="0">
      <Logger className="org.apache.catalina.logger.FileLogger"
              prefix="catalina_log." suffix=".txt"
              timestamp="true"/>
      <Realm className="org.apache.catalina.realm.MemoryRealm" />
      <Host name="localhost" debug="0" appBase="webapps" unpackWARs="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve"
                 directory="logs"  prefix="localhost_access_log." suffix=".txt"
                 pattern="common"/>
        <Logger className="org.apache.catalina.logger.FileLogger"
                 directory="logs"  prefix="localhost_log." suffix=".txt"
                timestamp="true"/>
      </Host>
    </Engine>
  </Service>
  <Service name="Tomcat-Apache">
    <Engine className="org.apache.catalina.connector.warp.WarpEngine"
     name="Apache" debug="0">
      <Logger className="org.apache.catalina.logger.FileLogger"
              prefix="apache_log." suffix=".txt"
              timestamp="true"/>
      <Realm className="org.apache.catalina.realm.MemoryRealm" />
    </Engine>
  </Service>
</Server>
----------



---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Mime
View raw message