tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <>
Subject Re: Background thread died; no errors in log; invoking backgroundProcess via JMX has no effect
Date Thu, 05 Jun 2014 19:06:54 GMT
Hash: SHA256


On 6/5/14, 11:58 AM, Konstantin Kolinko wrote:
> 2014-06-05 19:19 GMT+04:00 Christopher Schultz
> <>:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>> All,
>> One of dev instances this morning is screaming because I have
>> hundreds of active sessions. I checked, and it looks like the
>> background processor thread for the context is not running,
>> therefore no sessions are expiring.
>> Tomcat 7.0.52, Debian Linux x64, Oracle Java 1.7.0_55
>> Here are the still-running threads:
>> "catalina-exec-946" "Attach Listener" "MySQL Statement
>> Cancellation Timer" "catalina-exec-338" "catalina-exec-337" 
>> "catalina-exec-5" "catalina-exec-4" "catalina-exec-3" 
>> "http-nio-" 
>> "http-nio-" 
>> "http-nio-" 
>> "ajp-bio-8215-AsyncTimeout" "ajp-bio-8215-Acceptor-0" "Memcached
>> IO over {MemcachedConnection to localhost/}" 
>> "Abandoned connection cleanup thread" "Thread-3" 
>> "NioBlockingSelector.BlockPoller-1" "GC Daemon" "Service Thread" 
>> "C2 CompilerThread1" "C2 CompilerThread0" "Signal Dispatcher" 
>> "Finalizer" "Reference Handler" "main" "VM Thread" "GC task
>> thread#0 (ParallelGC)" "GC task thread#1 (ParallelGC)" "VM
>> Periodic Task Thread"
>> I tried invoking "backgroundProcess" on my Manager bean via JMX
>> but it didn't seem to actually clean anything up. Is there any
>> hope of recovery without bouncing the web application?
>> Also, it would be nice to get a notification if the thread is
>> dying due to some exception. I searched-though catalina.out and
>> found nothing relevant. Any suggestions?
> In general, dying threads are handled via 
> Thread.setUncaughtExceptionHandler(...) and 
> ThreadGroup.uncaughtException(...)
> The default behaviour in JRE it so log to System.err. So
> catalina.out is the place where I expect it to be written.

That's what I would have thought, too. I didn't see anything,

> (It is not nice as it bypasses logging framework, but that is how
> it is currently).
> A unusual culprit is OutOfMemoryError.

I think you mean "a usual culprit" and I agree: this has happened in
the past where the background processor thread dies due to OOME.

>> I tried invoking "backgroundProcess" on my Manager bean via JMX
>> but it didn't seem to actually clean anything up. Is there any
>> hope of recovery without bouncing the web application?
> The background thread is started via threadStart() which is called 
> from ContainerBase.startInternal() /
> StandardContext.startInternal().
> So I see no hope of recovering unless the container that owns the 
> thread is stopped/started.  I think that by default the thread is 
> started by Engine, so Engine needs to be bounced here.

The JMX method "backgroundProcess" should call
ManagerBase.backgroundProcess which should go ahead and expire the
sessions. I don't see a reason why it wouldn't work. Unfortunately,
Tomcat's Manager wasn't in debug-log mode or I would have been able to
see what happened when I invoked backgroundProcess.

> BTW, javadoc for ContainerBase.backgroundProcess() says "This
> method will be invoked inside the classloading context of this
> container."  I think that is not true when it is being invoked via
> JMX.

This is ManagerBase.backgroundProcess, which does not have the same
documentation. While ContainerBase.backgroundProcess would usually
invoke ManagerBase.backgroundProcess, using JMX goes directly to

I'm still curious why invoking ManagerBase.backgroundProcess via JMX
had no effect. I'm not sure what the "count" and
"processExpiresFrequency" values were at the time, so my calls could
have been rejected due to background-processing-throttling.

- -chris
Version: GnuPG v1
Comment: GPGTools -
Comment: Using GnuPG with Thunderbird -


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message