tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Freestone <tefreest...@gmail.com>
Subject Strange tomcat error and running out of file handles
Date Mon, 24 Jun 2013 20:32:32 GMT
Hello,
We are running tomcat 6.0.36 (java build 1.6.0_31) on REHL5.  It works most
of time except recently we started running out of files.  Here is the error
I am getting:


Exception in thread "pool-1-thread-1" java.lang.IllegalMonitorStateException
        at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
        at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:406)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
Exception in thread "pool-1-thread-2" java.lang.IllegalMonitorStateException
        at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
        at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:406)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
Jun 22, 2013 8:18:55 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@378186dc last
access:2013-06-22 20:18:50.645
Jun 22, 2013 8:19:00 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@6ec71370 last
access:2013-06-22 20:18:55.649
Jun 22, 2013 8:19:00 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@378186dc last
access:2013-06-22 20:18:55.646
Jun 22, 2013 8:19:05 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@6ec71370 last
access:2013-06-22 20:19:00.648
Jun 22, 2013 8:19:05 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@71312bbd last
access:2013-06-22 20:19:00.649
SEVERE: Unable to process request in NioReceiver
java.lang.ThreadDeath
        at java.lang.Thread.stop(Thread.java:758)
        at net.bull.javamelody.Action.stopThread(Action.java:388)
        at net.bull.javamelody.Action.killThread(Action.java:374)
        at net.bull.javamelody.Action.execute(Action.java:227)
        at
net.bull.javamelody.MonitoringController.executeActionIfNeeded(MonitoringController.java:117)
        at
net.bull.javamelody.MonitoringController.doActionIfNeededAndReport(MonitoringController.java:130)
        at
net.bull.javamelody.MonitoringFilter.doMonitoring(MonitoringFilter.java:340)
        at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:170)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:227)
        at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at
org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:448)
        at
org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:403)
        at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1703)
        at java.lang.Thread.run(Thread.java:662)
Jun 22, 2013 8:19:55 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
SEVERE: Unexpected poller error
java.lang.ThreadDeath
        at java.lang.Thread.stop(Thread.java:758)
        at net.bull.javamelody.Action.stopThread(Action.java:388)
        at net.bull.javamelody.Action.killThread(Action.java:374)
        at net.bull.javamelody.Action.execute(Action.java:227)
        at
net.bull.javamelody.MonitoringController.executeActionIfNeeded(MonitoringController.java:117)
        at
net.bull.javamelody.MonitoringController.doActionIfNeededAndReport(MonitoringController.java:130)
        at
net.bull.javamelody.MonitoringFilter.doMonitoring(MonitoringFilter.java:340)
        at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:170)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:227)
        at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at
org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:448)
        at
org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:403)
        at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1703)
        at java.lang.Thread.run(Thread.java:662)
Jun 22, 2013 8:20:00 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts

SEVERE: Unexpected poller error
java.lang.ThreadDeath
        at java.lang.Thread.stop(Thread.java:758)
        at net.bull.javamelody.Action.stopThread(Action.java:388)
        at net.bull.javamelody.Action.killThread(Action.java:374)
        at net.bull.javamelody.Action.execute(Action.java:227)
        at
net.bull.javamelody.MonitoringController.executeActionIfNeeded(MonitoringController.java:117)
        at
net.bull.javamelody.MonitoringController.doActionIfNeededAndReport(MonitoringController.java:130)
        at
net.bull.javamelody.MonitoringFilter.doMonitoring(MonitoringFilter.java:340)
        at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:170)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:227)
        at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at
org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:448)
        at
org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:403)
        at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1703)
        at java.lang.Thread.run(Thread.java:662)
Jun 22, 2013 8:20:00 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts

Then I get a bunch of
Exception in thread
"org%002ejasig%002ecas%002eticket%002e%0054icket%0047ranting%0054icket.data"
java.lang.IllegalMonitorStateException
        at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
        at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:176)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)


Eventually, the filesystem fills up and tomcat is dead.  Any ideas of why
we are getting the thread errors.  Thanks in advance!

tom
-- 

*****************************
Tom Freestone
(tefreestone@gmail.com)
*****************************

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message