incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <>
Subject Re: normal thread counts?
Date Mon, 29 Apr 2013 20:31:22 GMT
>  I used JMX to check current number of threads in a production cassandra machine, and
it was ~27,000.
That does not sound too good. 

My first guess would be lots of client connections. What client are you using, does it do
connection pooling ?
See the comments in cassandra.yaml around rpc_server_type, the default uses sync uses one
thread per connection, you may be better with HSHA. But if your app is leaking connection
you should probably deal with that first. 


Aaron Morton
Freelance Cassandra Consultant
New Zealand


On 30/04/2013, at 3:07 AM, William Oberman <> wrote:

> Hi,
> I'm having some issues.  I keep getting:
> ------------
> ERROR [GossipStage:1] 2013-04-28 07:48:48,876 (line 135)
Exception in thread Thread[GossipStage:1,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> --------------
> after a day or two of runtime.  I've checked and my system settings seem acceptable:
> memlock=unlimited
> nofiles=100000
> nproc=122944
> I've messed with heap sizes from 6-12GB (15 physical, m1.xlarge in AWS), and I keep OOM'ing
with the above error.
> I've found some (what seem to me) to be obscure references to the stack size interacting
with # of threads.  If I'm understanding it correctly, to reason about Java mem usage I have
to think of OS + Heap as being locked down, and the stack gets the "leftovers" of physical
memory and each thread gets a stack.
> For me, the system ulimit setting on stack is 10240k (no idea if java sees or respects
this setting).  My -Xss for cassandra is the default (I hope, don't remember messing with
it) of 180k.  I used JMX to check current number of threads in a production cassandra machine,
and it was ~27,000.  Is that a normal thread count?  Could my OOM be related to stack + number
of threads, or am I overlooking something more simple?
> will

View raw message