tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Addison <marvin.addi...@gmail.com>
Subject Re: Excessive CPU w/APR Connectors on tomcat-native 1.1.22
Date Thu, 12 Jan 2012 16:51:23 GMT
> Can you confirm whether or not the issue exists with 6.0.26 and 1.1.22?

I cannot.  We have tried repeatedly to reproduce this problem in a
test environment where such experimentation is tolerated, but the
problem simply does not manifest using available load testing tools.
We attempted to try 7.0.23+1.2.20 in production, but I couldn't get a
working configuration for the supported SSL/TLS protocols we require.

> How long do the periods of high CPU usage last?

Varies quite a bit.  Based on data from our enterprise monitoring
system, the problem lasts as little as 5 min or less and has lasted as
much as 25 min.  YourKit data from the incidents that have happened in
the past 24h shows similarly wide variation:

2m40s
1m38s
13m11s
8s

> How sure are you that the CPU is being burned in the threads that are in
> sendbb?

I'm certain of nothing, but I can share the evidence.  YourKit CPU
profiling data shows that request processing threads consume the
majority of resources during these periods.  No surprise there.
However, the only method that is consistently executed during these
periods is sendbb() by two or more threads.  For the last problem
period, which happens to be the shortest (8s), catalina-exec threads
accounted for 89% of CPU use.  Following are the RUNNABLE threads at
1s sample intervals during that period:

Stacks at 09:11:01 AM (uptime 1 day 3:26:47)
1. catalina-exec-45 [RUNNABLE, IN_NATIVE] CPU time: 1:57
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:32
org.apache.tomcat.jni.Socket.sendbb(long, int, int)

Stacks at 09:11:02 AM (uptime 1 day 3:26:48)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:21
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:33
org.apache.tomcat.jni.Socket.sendbb(long, int, int)

Stacks at 09:11:03 AM (uptime 1 day 3:26:49)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:22
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:34
org.apache.tomcat.jni.Socket.sendbb(long, int, int)

Stacks at 09:11:04 AM (uptime 1 day 3:26:50)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:23
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:35
org.apache.tomcat.jni.Socket.sendbb(long, int, int)

Stacks at 09:11:05 AM (uptime 1 day 3:26:51)
1. catalina-exec-38 [RUNNABLE, IN_NATIVE] CPU time: 1:28
java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int)
2. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:24
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
3. catalina-exec-44 [RUNNABLE] CPU time: 1:43
java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int)
4. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:35
org.apache.tomcat.jni.Socket.sendbb(long, int, int)

Stacks at 09:11:06 AM (uptime 1 day 3:26:52)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:25
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:36
org.apache.tomcat.jni.Socket.sendbb(long, int, int)

Stacks at 09:11:07 AM (uptime 1 day 3:26:53)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:25
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:37
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
3. catalina-exec-52 [RUNNABLE] CPU time: 0:04
java.lang.Throwable.fillInStackTrace()

Stacks at 09:11:08 AM (uptime 1 day 3:26:54)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:26
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:38
org.apache.tomcat.jni.Socket.sendbb(long, int, int)

Stacks at 09:11:09 AM (uptime 1 day 3:26:55)
1. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:39
org.apache.tomcat.jni.Socket.sendbb(long, int, int)

After the last sample, CPU usage drops precipitously.  Our application
is a Web SSO product that writes relatively small response payloads.
It's hard to explain two request processing threads that take several
seconds to write small response payloads without citing a bug in the
servlet container.

M

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message