tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Mikusa <dmik...@gopivotal.com>
Subject Re: Strange tomcat error and running out of file handles
Date Mon, 24 Jun 2013 21:09:10 GMT
On Jun 24, 2013, at 4:32 PM, Tom Freestone <tefreestone@gmail.com> wrote:

> Hello,
> We are running tomcat 6.0.36 (java build 1.6.0_31) on REHL5.  It works most
> of time except recently we started running out of files.  Here is the error
> I am getting:
> 
> 
> Exception in thread "pool-1-thread-1" java.lang.IllegalMonitorStateException
>        at
> java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>        at
> java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>        at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:406)
>        at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:662)
> Exception in thread "pool-1-thread-2" java.lang.IllegalMonitorStateException
>        at
> java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>        at
> java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>        at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:406)
>        at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:662)
> Jun 22, 2013 8:18:55 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@378186dc last
> access:2013-06-22 20:18:50.645
> Jun 22, 2013 8:19:00 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@6ec71370 last
> access:2013-06-22 20:18:55.649
> Jun 22, 2013 8:19:00 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@378186dc last
> access:2013-06-22 20:18:55.646
> Jun 22, 2013 8:19:05 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@6ec71370 last
> access:2013-06-22 20:19:00.648
> Jun 22, 2013 8:19:05 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@71312bbd last
> access:2013-06-22 20:19:00.649

> SEVERE: Unable to process request in NioReceiver
> java.lang.ThreadDeath
>        at java.lang.Thread.stop(Thread.java:758)
>        at net.bull.javamelody.Action.stopThread(Action.java:388)
>        at net.bull.javamelody.Action.killThread(Action.java:374)
>        at net.bull.javamelody.Action.execute(Action.java:227)
>        at
> net.bull.javamelody.MonitoringController.executeActionIfNeeded(MonitoringController.java:117)
>        at
> net.bull.javamelody.MonitoringController.doActionIfNeededAndReport(MonitoringController.java:130)
>        at
> net.bull.javamelody.MonitoringFilter.doMonitoring(MonitoringFilter.java:340)
>        at
> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:170)

Looks like your application is calling Thread.stop().  Who or what is making the requests
that trigger this exception? Have you looked in the access log to see what is sending these
requests?

<snip>

> 
> Then I get a bunch of
> Exception in thread
> "org%002ejasig%002ecas%002eticket%002e%0054icket%0047ranting%0054icket.data"
> java.lang.IllegalMonitorStateException
>        at
> java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
>        at
> java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>        at java.util.concurrent.DelayQueue.take(DelayQueue.java:176)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>        at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:662)
> 
> 
> Eventually, the filesystem fills up and tomcat is dead.  

What is filling up the file system?  Is it this log file?

> Any ideas of why we are getting the thread errors. 

The stack trace above seems to indicate that someone is instructing Java Melody, a monitoring
tool, to stop particular threads.  My thought would be to figure what is triggering that to
happen (see above about looking at the access log), stop it and then continue troubleshooting
from there.

Dan



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message