cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Desimpel, Ignace" <>
Subject Hanging after OutOfMemory exception
Date Mon, 18 Feb 2013 12:44:49 GMT
Running Cassandra as an embedded service (not really related I think), and due to an out-of-memory
exception ( 'wrong' configuration from my part ), I could make the system hang although I
would have liked it to stop running.

Below I copied pieces of the stack trace that are related in my opinion. Attached is a file
with all parameters (java, machine, libs, ...) and the stack trace.
Hopefully this helps make the code better (or my code ...)


Here is my opinion :

Thread-174 is processing a System.exit() due to the uncaughtException handling in CassandraDaemon.
That System.exit() gets a lock and runs the ApplicationShutdownHooks.

The StorageService has a shutdownhook called StorageServiceShutdownHook. That hook is calling
CommitLog.instance.shutdownBlocking(), and that routine is (in my case) calling shutdown()
and awaitTermination() on a PeriodicCommitlogExecutorService. Therefore it is waiting on a
thread called COMMIT-LOG-WRITER to terminate.
But that COMMIT-LOG-WRITER also has got an uncaughtException, and in response to that it is
calling System.exit(). But that call cannot return since "Thread-174" is already holding a

So the system is "hanging" on the System.exit due to a (multiple) OutOfMemory exception

"StorageServiceShutdownHook" Id=41 WAITING on java.lang.Thread@79dc2ba2
                at java.lang.Object.wait(Native Method)
                -  waiting on java.lang.Thread@79dc2ba2
                at java.lang.Thread.join(
                at java.lang.Thread.join(
                at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService.awaitTermination(
                at org.apache.cassandra.db.commitlog.CommitLog.shutdownBlocking(
                at org.apache.cassandra.service.StorageService$1.runMayThrow(

"Commitlog Shutdown" Id=22868 TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$4.runMayThrow(

"COMMIT-LOG-WRITER" Id=39 BLOCKED on java.lang.Class@1cc7b00c owned by "Thread-174" Id=242
                at java.lang.Shutdown.exit(
                -  blocked on java.lang.Class@1cc7b00c
                at java.lang.Runtime.exit(
                at java.lang.System.exit(
                at org.apache.cassandra.service.CassandraDaemon$1.uncaughtException(
                at java.lang.ThreadGroup.uncaughtException(
                at java.lang.ThreadGroup.uncaughtException(
                at java.lang.Thread.dispatchUncaughtException(

"Thread-174" Id=242 WAITING on java.lang.Thread@458e439a
                at java.lang.Object.wait(Native Method)
                -  waiting on java.lang.Thread@458e439a
                at java.lang.Thread.join(
                at java.lang.Thread.join(
                at java.lang.ApplicationShutdownHooks.runHooks(
                at java.lang.ApplicationShutdownHooks$
                at java.lang.Shutdown.runHooks(
                at java.lang.Shutdown.sequence(
                at java.lang.Shutdown.exit(
                -  locked java.lang.Class@1cc7b00c

View raw message