incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: normal thread counts?
Date Mon, 29 Apr 2013 20:31:22 GMT
>  I used JMX to check current number of threads in a production cassandra machine, and
it was ~27,000.
That does not sound too good. 

My first guess would be lots of client connections. What client are you using, does it do
connection pooling ?
See the comments in cassandra.yaml around rpc_server_type, the default uses sync uses one
thread per connection, you may be better with HSHA. But if your app is leaking connection
you should probably deal with that first. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 30/04/2013, at 3:07 AM, William Oberman <oberman@civicscience.com> wrote:

> Hi,
> 
> I'm having some issues.  I keep getting:
> ------------
> ERROR [GossipStage:1] 2013-04-28 07:48:48,876 AbstractCassandraDaemon.java (line 135)
Exception in thread Thread[GossipStage:1,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> --------------
> after a day or two of runtime.  I've checked and my system settings seem acceptable:
> memlock=unlimited
> nofiles=100000
> nproc=122944
> 
> I've messed with heap sizes from 6-12GB (15 physical, m1.xlarge in AWS), and I keep OOM'ing
with the above error.
> 
> I've found some (what seem to me) to be obscure references to the stack size interacting
with # of threads.  If I'm understanding it correctly, to reason about Java mem usage I have
to think of OS + Heap as being locked down, and the stack gets the "leftovers" of physical
memory and each thread gets a stack.
> 
> For me, the system ulimit setting on stack is 10240k (no idea if java sees or respects
this setting).  My -Xss for cassandra is the default (I hope, don't remember messing with
it) of 180k.  I used JMX to check current number of threads in a production cassandra machine,
and it was ~27,000.  Is that a normal thread count?  Could my OOM be related to stack + number
of threads, or am I overlooking something more simple?
> 
> will
> 


Mime
View raw message