incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srmore <comom...@gmail.com>
Subject Error during startup - java.lang.OutOfMemoryError: unable to create new native thread
Date Tue, 10 Sep 2013 03:15:38 GMT
I have a 5 node cluster with a load of around 300GB each. A node went down
and does not come up. I can see the following exception in the logs.

ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line
139) Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at
java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
        at
java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1392)
        at
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:77)
        at
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:65)
        at
org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor.<init>(JMXConfigurableThreadPoolExecutor.java:34)
        at
org.apache.cassandra.concurrent.StageManager.multiThreadedConfigurableStage(StageManager.java:68)
        at
org.apache.cassandra.concurrent.StageManager.<clinit>(StageManager.java:42)
        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:344)
        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:173)

The *ulimit -u* output is
*515042*

Which is far more than what is recommended [1] (10240) and I am skeptical
to set it to unlimited as recommended here [2]

Any pointers as to what could be the issue and how to get the node up.



[1]
http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=install/recommended_settings#cassandra/install/installRecommendSettings.html

[2]
http://mail-archives.apache.org/mod_mbox/cassandra-user/201303.mbox/%3CCAPqEvGE474Omea1BFLJ6U_pbAkOwWxk=Dwo35_pc-ATwB4_6iA@mail.gmail.com%3E

Thanks !

Mime
View raw message