incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Oberman <ober...@civicscience.com>
Subject normal thread counts?
Date Mon, 29 Apr 2013 15:07:44 GMT
Hi,

I'm having some issues.  I keep getting:
------------
ERROR [GossipStage:1] 2013-04-28 07:48:48,876 AbstractCassandraDaemon.java
(line 135) Exception in thread Thread[GossipStage:1,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
--------------
after a day or two of runtime.  I've checked and my system settings seem
acceptable:
memlock=unlimited
nofiles=100000
nproc=122944

I've messed with heap sizes from 6-12GB (15 physical, m1.xlarge in AWS),
and I keep OOM'ing with the above error.

I've found some (what seem to me) to be obscure references to the stack
size interacting with # of threads.  If I'm understanding it correctly, to
reason about Java mem usage I have to think of OS + Heap as being locked
down, and the stack gets the "leftovers" of physical memory and each thread
gets a stack.

For me, the system ulimit setting on stack is 10240k (no idea if java sees
or respects this setting).  My -Xss for cassandra is the default (I hope,
don't remember messing with it) of 180k.  I used JMX to check current
number of threads in a production cassandra machine, and it was ~27,000.
 Is that a normal thread count?  Could my OOM be related to stack + number
of threads, or am I overlooking something more simple?

will

Mime
View raw message