Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: error (athena.apache.org: local policy)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: Strange tomcat error and running out of file handles
From: Daniel Mikusa <dmikusa@gopivotal.com>
In-Reply-To: 
 <CALkfsc8kiiH80poeXh5JGPbT36O=wf9MaW408HLse8aETayV4A@mail.gmail.com>
Date: Mon, 24 Jun 2013 17:09:10 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <6F83D93F-8D66-4FDF-90EC-1310B995F28C@gopivotal.com>
References: 
 <CALkfsc8kiiH80poeXh5JGPbT36O=wf9MaW408HLse8aETayV4A@mail.gmail.com>
To: Tomcat Users List <users@tomcat.apache.org>

On Jun 24, 2013, at 4:32 PM, Tom Freestone <tefreestone@gmail.com> =
wrote:

> Hello,
> We are running tomcat 6.0.36 (java build 1.6.0_31) on REHL5.  It works =
most
> of time except recently we started running out of files.  Here is the =
error
> I am getting:
>=20
>=20
> Exception in thread "pool-1-thread-1" =
java.lang.IllegalMonitorStateException
>        at
> =
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.jav=
a:127)
>        at
> =
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueu=
edSynchronizer.java:1239)
>        at
> =
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>        at
> =
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:406=
)
>        at
> =
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:94=
7)
>        at
> =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:907)
>        at java.lang.Thread.run(Thread.java:662)
> Exception in thread "pool-1-thread-2" =
java.lang.IllegalMonitorStateException
>        at
> =
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.jav=
a:127)
>        at
> =
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueu=
edSynchronizer.java:1239)
>        at
> =
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>        at
> =
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:406=
)
>        at
> =
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:94=
7)
>        at
> =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:907)
>        at java.lang.Thread.run(Thread.java:662)
> Jun 22, 2013 8:18:55 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for =
the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@378186dc =
last
> access:2013-06-22 20:18:50.645
> Jun 22, 2013 8:19:00 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for =
the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@6ec71370 =
last
> access:2013-06-22 20:18:55.649
> Jun 22, 2013 8:19:00 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for =
the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@378186dc =
last
> access:2013-06-22 20:18:55.646
> Jun 22, 2013 8:19:05 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for =
the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@6ec71370 =
last
> access:2013-06-22 20:19:00.648
> Jun 22, 2013 8:19:05 PM
> org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
> WARNING: Channel key is registered, but has had no interest ops for =
the
> last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@71312bbd =
last
> access:2013-06-22 20:19:00.649

> SEVERE: Unable to process request in NioReceiver
> java.lang.ThreadDeath
>        at java.lang.Thread.stop(Thread.java:758)
>        at net.bull.javamelody.Action.stopThread(Action.java:388)
>        at net.bull.javamelody.Action.killThread(Action.java:374)
>        at net.bull.javamelody.Action.execute(Action.java:227)
>        at
> =
net.bull.javamelody.MonitoringController.executeActionIfNeeded(MonitoringC=
ontroller.java:117)
>        at
> =
net.bull.javamelody.MonitoringController.doActionIfNeededAndReport(Monitor=
ingController.java:130)
>        at
> =
net.bull.javamelody.MonitoringFilter.doMonitoring(MonitoringFilter.java:34=
0)
>        at
> =
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:170)

Looks like your application is calling Thread.stop().  Who or what is =
making the requests that trigger this exception? Have you looked in the =
access log to see what is sending these requests?

<snip>

>=20
> Then I get a bunch of
> Exception in thread
> =
"org%002ejasig%002ecas%002eticket%002e%0054icket%0047ranting%0054icket.dat=
a"
> java.lang.IllegalMonitorStateException
>        at
> =
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.jav=
a:127)
>        at
> =
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueu=
edSynchronizer.java:1239)
>        at
> =
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
>        at java.util.concurrent.DelayQueue.take(DelayQueue.java:176)
>        at
> =
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch=
eduledThreadPoolExecutor.java:609)
>        at
> =
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch=
eduledThreadPoolExecutor.java:602)
>        at
> =
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:94=
7)
>        at
> =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:907)
>        at java.lang.Thread.run(Thread.java:662)
>=20
>=20
> Eventually, the filesystem fills up and tomcat is dead. =20

What is filling up the file system?  Is it this log file?

> Any ideas of why we are getting the thread errors.=20

The stack trace above seems to indicate that someone is instructing Java =
Melody, a monitoring tool, to stop particular threads.  My thought would =
be to figure what is triggering that to happen (see above about looking =
at the access log), stop it and then continue troubleshooting from =
there.

Dan


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org