tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Bradford <netg...@ipudev.com>
Subject Re: Load Balanced Tomcat freezing under high load
Date Thu, 27 Feb 2003 16:00:22 GMT
   Hi Kwong.. thanks for your post... I did some experimenting last night 
and think I found that my problem was garbage collection; it was the 
garbage collector process that was going haywire and consuming all the 
CPU. Starting Tomcat with a lower heap size (I had over-allocated 
slightly and it was starting to use swap) and using -Xincgc in JAVA_OPTS 
seems to have helped significantly, although I've traded off CPU usage, 
which is now quite a bit higher, but manageable.

   Thanks!
   -Mike

On Wednesday, February 26, 2003, at 04:53 PM, Kwong Chong wrote:

> Hi,
>
> this problem will probably be due partly to the round robin load
> balancing in the modjk2 connector, as well as the application trying to
> process too many requests at one time (context switching), if you are
> using the sun jdk1.4, that might cause problems as well.
>
> With round robin load balancing, as soon as you start receiving load
> faster than you can process them, you will start seeing problems where
> the slower workers tend to receive more and more requests in their
> queue; the more requests they try to process at once, the slower they
> become (due to context switching etc), and the slower they become the
> longer their queue gets, etc etc.
>
> One way to get around this might be to set limits on tomcat (tomcat
> experts help me out here) lower the setting on maxprocessors and the
> accept count so that the worker will reject new settings before it gets
> more than it can handle.
>
> I have also experienced a problem using the sun jdk1.4 where
> occasionally a worker will go into a state where it just about uses up
> as much cpu as the system will allow it and stop responding - checking
> out the process shows it makes a HUGE number of system calls;
> truss would show a lot of
>
> lwp_suspend
> lwp_continue
> lwp_kill
> lwp_mutex_wakeup
> lwp_mutex_lock
> lwp_cond_wait
> lwp_cond_signal
> lwp_cond_broadcast
>
> I've yet to get to the bottom of this ... but when this happens nothing
> short of a restart of the process seems to help.
>
> Hope this helps,
>
> Kwong
>
>
> -----Original Message-----
> From: netguru@ipudev.com [mailto:netguru@ipudev.com]
> Sent: Thursday, 27 February 2003 7:35
> To: tomcat-user@jakarta.apache.org
> Subject: Load Balanced Tomcat freezing under high load
>
>
>
>   Hi all.. we're running Apache 2.0.44, mod_jk2 2.0.2 and Tomcat 4.0.6 
> (all
> compiled from source) load-balanced across 4 machines. Occasionally
> (usually at peak times) one of the systems (usually the first one listed
> in the workers2.properties) will freeze up and the CPU will go through 
> the
> roof. 'top' shows that its one particular java process consuming 99% of
> the CPU. (I think it's one of the first ones spawned when tomcat is
> started). At this point the Apache hits it's MaxClients (512) and the 
> site
> goes down, despite the other Tomcat workers being alive and well. I'm
> pretty sure it's not an Apache problem as I just upgraded from
> 1.3.27/mod_jk-1.2 last night and the problem has happened again twice
> today.
>
>   I'm getting a lot of these in the Apache error log:
> --------------- error_log
> [Wed Feb 26 01:22:47 2003] [error] jk2_init() Can't find child 26949 in
> scoreboard
> [Wed Feb 26 01:22:47 2003] [error] mod_jk child workerEnv in error 
> state 3
> [Wed Feb 26 01:22:47 2003] [error] mod_jk child init 1 -2
> [Wed Feb 26 01:22:49 2003] [error] mod_jk child workerEnv in error 
> state 3
> [Wed Feb 26 01:22:50 2003] [error] mod_jk child workerEnv in error 
> state 3
> --------------- etc..etc
>   and a few of these when the site got busier:
> ---------------
> [Wed Feb 26 10:30:04 2003] [error] Error ajp_process_callback - write 
> failed
> [Wed Feb 26 10:30:04 2003] [error] ajp13.service() ajpGetReply
> recoverable error 3
> [Wed Feb 26 10:30:04 2003] [error] Error ajp_process_callback - write 
> failed
> [Wed Feb 26 10:30:04 2003] [error] ajp13.service() ajpGetReply
> recoverable error 3
> [Wed Feb 26 10:30:04 2003] [error] ajp13.service() Error  forwarding
> ajp13:jvm1:8009 1 0
> [Wed Feb 26 10:30:04 2003] [error] mod_jk.handler() Error connecting to
> tomcat 3
> [Wed Feb 26 10:30:13 2003] [error] mod_jk child init 1 0
> ---------- etc etc..
>   I'm also seeing a lot of these in the catalina_log on the Tomcat
> workers:
> ----------
> 2003-02-26 09:50:51 Ajp13Processor[8009][133] process: invoke
> java.net.SocketException: Socket closed
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>         at 
> java.net.SocketOutputStream.write(SocketOutputStream.java:126)
>         at org.apache.ajp.Ajp13.send(Ajp13.java:525)
>         at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
>         at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
>         at
> org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
>         at
> org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
>         at
> org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
>         at java.lang.Thread.run(Thread.java:536)
> ---------
>
>   Beyond that, the logs don't tell me much. Anyway, sorry for the huge
> length of this e-mail; if anyone has any suggestions it would be a huge
> help.
>
>   Thanks!
>   -Mike
>    netguru at ipudev dot com
>
>
> -------- workers2.properties
> [shm]
> file=/usr/local/apache/logs/shm.file
> size=4194304
>
> [channel.socket:jvm2:8009]
> port=8009
> host=192.168.0.31
> group=lb
> lb_factor=1
> disabled=0
> route=jvm2
>
> [channel.socket:jvm1:8009]
> port=8009
> host=192.168.0.30
> group=lb
> lb_factor=1
> disabled=0
> route=jvm1
>
> [channel.socket:jvm3:8009]
> port=8009
> host=192.168.0.32
> group=lb
> lb_factor=1
> disabled=0
> route=jvm3
>
> [channel.socket:jvm4:8009]
> port=8009
> host=192.168.0.33
> group=lb
> lb_factor=1
> disabled=0
> route=jvm4
>
> [various uri mappings to lb worker...]
> ---------
> --------- server.xml (same on all Tomcat servers)
> <Server port="8005" shutdown="SHUTDOWN" debug="0">
>   <Service name="Tomcat-Standalone">
>     <Connector className="org.apache.ajp.tomcat4.Ajp13Connector"
>                port="8009" minProcessors="5" maxProcessors="500"
>                acceptCount="100" debug="0"/>
>     <!-- jvmRoute is configured appropriately for each system -->
>     <Engine jvmRoute="jvm1" name="Standalone" defaultHost="localhost"
> debug="0">
>       <Logger className="org.apache.catalina.logger.FileLogger"
>               prefix="catalina_log." suffix=".txt"
>               timestamp="true"/>
>       <Realm className="org.apache.catalina.realm.MemoryRealm" />
>       <Host name="localhost" debug="0" appBase="webapps" 
> unpackWARs="true">
>         <Valve className="org.apache.catalina.valves.AccessLogValve"
>                  directory="logs"  prefix="localhost_access_log."
> suffix=".txt"
>                  pattern="common"/>
>         <Logger className="org.apache.catalina.logger.FileLogger"
>                  directory="logs"  prefix="localhost_log." suffix=".txt"
>                 timestamp="true"/>
>       </Host>
>     </Engine>
>   </Service>
>   <Service name="Tomcat-Apache">
>     <Engine className="org.apache.catalina.connector.warp.WarpEngine"
>      name="Apache" debug="0">
>       <Logger className="org.apache.catalina.logger.FileLogger"
>               prefix="apache_log." suffix=".txt"
>               timestamp="true"/>
>       <Realm className="org.apache.catalina.realm.MemoryRealm" />
>     </Engine>
>   </Service>
> </Server>
> ----------
>
>
> ----------------
> Powered by telstra.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Mime
View raw message