tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benimaur Gao <benim...@gmail.com>
Subject All worker threads of my tomcat have been occupied!
Date Thu, 02 Jan 2014 10:20:16 GMT
hi, all

I found my tomcat refusing to work this morning.
I tried jstack to get some info, and then I found a lot of thread call
stack like:

"http-8082-154" daemon prio=10 tid=0x00007f711c21f800 nid=0x5b0a waiting on
> condition [0x00007f70dc887000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:132)
>     at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>     at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>     at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>     at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>     at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
>     at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
>     at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>     at java.lang.Thread.run(Thread.java:662)
>


Actually, when I used grep to get more clue
# grep -B5 -A7 org.apache.catalina.valves.ErrorReportValve.invoke 14446.dump

I got:

Dump File: 14446.dump.1388636053
> execute external commands: ^[[01;31mgrep -B5 -A7
> org.apache.catalina.valves.ErrorReportValve.invoke^[[00m
> "http-8082-200" daemon prio=10 tid=0x00007f711c14b000 nid=0x6a52 waiting
> on condition [0x00007f70d9a59000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:132)
>     at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>     at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>     at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>     at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>     at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
>     at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
>     at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>     at java.lang.Thread.run(Thread.java:662)
>
> "http-8082-199" daemon prio=10 tid=0x00007f711c148800 nid=0x69f3 waiting
> on condition [0x00007f70d9b5a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:132)
>     at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>     at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>     at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>     at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>     at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
>     at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
>     at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>     at java.lang.Thread.run(Thread.java:662)
>
> "http-8082-198" daemon prio=10 tid=0x00007f711c146000 nid=0x6992 waiting
> on condition [0x00007f70d9c5b000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:132)
>     at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>     at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>     at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>     at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>     at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
>     at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
>     at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>     at java.lang.Thread.run(Thread.java:662)
> ... ...
>
> "http-8082-2" daemon prio=10 tid=0x00007f711c002800 nid=0x38a4 waiting on
> condition [0x00007f7124b7a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:132)
>     at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>     at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>     at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>     at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>     at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
>     at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
>     at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>     at java.lang.Thread.run(Thread.java:662)
>
> "http-8082-1" daemon prio=10 tid=0x00007f711c001000 nid=0x38a2 waiting on
> condition [0x00007f71099f8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:132)
>     at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>     at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>     at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>     at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>     at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
>     at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
>     at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>     at java.lang.Thread.run(Thread.java:662)
>

It seems all of the 200 JIoEndpoint$Worker threads had been occupied. and I
guess that's why tomcat refuse to serve any incoming request.

After I got a copy of the source of my tomcat. I found
$ grep -B3 -A8 context.getPaused\(\)
/home/benimaur/workspace/eclipse/tomcat/src/org/apache/catalina/core/StandardContextValve.java

> 126-
> 127-        // Wait if we are reloading
> 128-        boolean reloaded = false;
> 129:        while (context.getPaused()) {
> 130-            reloaded = true;
> 131-            try {
> 132-                Thread.sleep(1000);
> 133-            } catch (InterruptedException e) {
> 134-                ;
> 135-            }
> 136-        }
> 137-
>

It seems context had been set to reload state at some time, but I can't
figure out under what  circumstance would put tomcat into such state.
does any one here could give me more clue? thanks in advance.

======================================

the version of my tomcat is 6.0.37

# java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

# uname -a
Linux his-web01 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message