Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4D766F00B for ; Wed, 1 May 2013 21:23:09 +0000 (UTC) Received: (qmail 64577 invoked by uid 500); 1 May 2013 21:23:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 64549 invoked by uid 500); 1 May 2013 21:23:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 64540 invoked by uid 99); 1 May 2013 21:23:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 May 2013 21:23:06 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: encountered temporary error during SPF processing of domain of oberman@civicscience.com) Received: from [209.85.210.52] (HELO mail-da0-f52.google.com) (209.85.210.52) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 May 2013 21:23:02 +0000 Received: by mail-da0-f52.google.com with SMTP id j17so813515dan.39 for ; Wed, 01 May 2013 14:22:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-originating-ip:in-reply-to:references :from:date:message-id:subject:to:content-type:x-gm-message-state; bh=gSv4wv33cLeJd6hIB9Yq2qd2l1DwwKHcbAkJTKQVghY=; b=a4CBzL+xluj3mNfpwakf2NaUiX1Q09RY0++beeKyWb1Ks+t5QuZGh6ig0zl7X2kz+d /JBdpiKBszmV/RCDviwNPge5Ad1IG3TJJV6OlDnR+5fp57DsWrRAjBJ5YFqK7wZ5zXGQ K9SSkQCSGXdcxlDOy0ZS3ZzNR6H3p6n0kZlMpn2GBNYeqLbsK6GocqZOJ15vZb6+4jTQ xaKwgpatW+8u6faj/pHug7qtjWo6gkf+2PqR4Ug6bCXxT4E/rpw3Pc0GIEYErJ75NFC5 OUqtwtQFavmKr+AvP6jgKpwejB3pV1SbENr1sgsKJgvEWjSEM2Y16M9LMefEEwEBt8pV uUhg== X-Received: by 10.66.8.138 with SMTP id r10mr6797594paa.55.1367443341724; Wed, 01 May 2013 14:22:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.245.66 with HTTP; Wed, 1 May 2013 14:22:01 -0700 (PDT) X-Originating-IP: [72.65.238.50] In-Reply-To: <87166F6C-71A5-4677-89F0-0E3765FBF6D0@ecyrd.com> References: <11010AD8-0A80-4345-845C-C2BF7AD9BE27@thelastpickle.com> <6A153C4D-3E72-4105-BE4D-D7C11FEE56E7@thelastpickle.com> <87166F6C-71A5-4677-89F0-0E3765FBF6D0@ecyrd.com> From: William Oberman Date: Wed, 1 May 2013 17:22:01 -0400 Message-ID: Subject: Re: normal thread counts? To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=bcaec51dd3c7a25eb804dbaeb61b X-Gm-Message-State: ALoCoQnbI8/DkQYtXRs1an27WZxmmcJ13iPlifxx/PuBv92Y5xuZPOVkeNQ6Mrk2gAVGzP7HyfaI X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51dd3c7a25eb804dbaeb61b Content-Type: text/plain; charset=ISO-8859-1 That has GOT to be it. 1.1.10 upgrade it is... On Wed, May 1, 2013 at 5:09 PM, Janne Jalkanen wrote: > > This sounds very much like > https://issues.apache.org/jira/browse/CASSANDRA-5175, which was fixed in > 1.1.10. > > /Janne > > On Apr 30, 2013, at 23:34 , aaron morton wrote: > > Many many many of the threads are trying to talk to IPs that aren't in > the cluster (I assume they are the IP's of dead hosts). > > Are these IP's from before the upgrade ? Are they IP's you expect to see ? > > Cross reference them with the output from nodetool gossipinfo to see why > the node thinks they should be used. > Could you provide a list of the thread names ? > > One way to remove those IPs that may be to rolling restart with > -Dcassandra.load_ring_state=false i the JVM opts at the bottom of > cassandra-env.sh > > The OutboundTcpConnection threads are created in pairs by the > OutboundTcpConnectionPool, which is created here > https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/MessagingService.java#L502 The > threads are created in the OutboundTcpConnectionPool constructor checking > to see if this could be the source of the leak. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 1/05/2013, at 2:18 AM, William Oberman > wrote: > > I use phpcassa. > > I did a thread dump. 99% of the threads look very similar (I'm using > 1.1.9 in terms of matching source lines). The thread names are all like > this: "WRITE-/10.x.y.z". There are a LOT of duplicates (in terms of the > same IP). Many many many of the threads are trying to talk to IPs that > aren't in the cluster (I assume they are the IP's of dead hosts). The > stack trace is basically the same for them all, attached at the bottom. > > There is a lot of things I could talk about in terms of my situation, but > what I think might be pertinent to this thread: I hit a "tipping point" > recently and upgraded a 9 node cluster from AWS m1.large to m1.xlarge > (rolling, one at a time). 7 of the 9 upgraded fine and work great. 2 of > the 9 keep struggling. I've replaced them many times now, each time using > this process: > http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node > And even this morning the only two nodes with a high number of threads are > those two (yet again). And at some point they'll OOM. > > Seems like there is something about my cluster (caused by the recent > upgrade?) that causes a thread leak on OutboundTcpConnection But I don't > know how to escape from the trap. Any ideas? > > > -------- > stackTrace = [ { > className = sun.misc.Unsafe; > fileName = Unsafe.java; > lineNumber = -2; > methodName = park; > nativeMethod = true; > }, { > className = java.util.concurrent.locks.LockSupport; > fileName = LockSupport.java; > lineNumber = 158; > methodName = park; > nativeMethod = false; > }, { > className = > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject; > fileName = AbstractQueuedSynchronizer.java; > lineNumber = 1987; > methodName = await; > nativeMethod = false; > }, { > className = java.util.concurrent.LinkedBlockingQueue; > fileName = LinkedBlockingQueue.java; > lineNumber = 399; > methodName = take; > nativeMethod = false; > }, { > className = org.apache.cassandra.net.OutboundTcpConnection; > fileName = OutboundTcpConnection.java; > lineNumber = 104; > methodName = run; > nativeMethod = false; > } ]; > ---------- > > > > > On Mon, Apr 29, 2013 at 4:31 PM, aaron morton wrote: > >> I used JMX to check current number of threads in a production cassandra >> machine, and it was ~27,000. >> >> That does not sound too good. >> >> My first guess would be lots of client connections. What client are you >> using, does it do connection pooling ? >> See the comments in cassandra.yaml around rpc_server_type, the default >> uses sync uses one thread per connection, you may be better with HSHA. But >> if your app is leaking connection you should probably deal with that first. >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 30/04/2013, at 3:07 AM, William Oberman >> wrote: >> >> Hi, >> >> I'm having some issues. I keep getting: >> ------------ >> ERROR [GossipStage:1] 2013-04-28 07:48:48,876 >> AbstractCassandraDaemon.java (line 135) Exception in thread >> Thread[GossipStage:1,5,main] >> java.lang.OutOfMemoryError: unable to create new native thread >> -------------- >> after a day or two of runtime. I've checked and my system settings seem >> acceptable: >> memlock=unlimited >> nofiles=100000 >> nproc=122944 >> >> I've messed with heap sizes from 6-12GB (15 physical, m1.xlarge in AWS), >> and I keep OOM'ing with the above error. >> >> I've found some (what seem to me) to be obscure references to the stack >> size interacting with # of threads. If I'm understanding it correctly, to >> reason about Java mem usage I have to think of OS + Heap as being locked >> down, and the stack gets the "leftovers" of physical memory and each thread >> gets a stack. >> >> For me, the system ulimit setting on stack is 10240k (no idea if java >> sees or respects this setting). My -Xss for cassandra is the default (I >> hope, don't remember messing with it) of 180k. I used JMX to check current >> number of threads in a production cassandra machine, and it was ~27,000. >> Is that a normal thread count? Could my OOM be related to stack + number >> of threads, or am I overlooking something more simple? >> >> will >> >> >> > > > > > > --bcaec51dd3c7a25eb804dbaeb61b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
That has GOT to be it. =A01.1.10 upgrade it is...


On Wed, May 1, 201= 3 at 5:09 PM, Janne Jalkanen <Janne.Jalkanen@ecyrd.com> wrote:
This sounds very much like=A0https://issues.apache.org/= jira/browse/CASSANDRA-5175, which was fixed in 1.1.10.

/Janne

On Apr 30, 2013, at = 23:34 , aaron morton <aaron@thelastpickle.com> wrote:

=A0Many many many of the threads are tryi= ng to talk to IPs that aren't in the cluster (I assume they are the IP&= #39;s of dead hosts).=A0
Are these IP's from before the upgrade ? Are they IP&= #39;s you expect to see ?=A0

Cross reference them = with the output from nodetool gossipinfo to see why the node thinks they sh= ould be used.=A0
Could you provide a list of the thread names ?=A0

= One way to remove those IPs that may be to rolling restart with -Dcassandra= .load_ring_state=3Dfalse i the JVM opts at the bottom of cassandra-env.sh

Cheers



I use phpca= ssa.

I did a thread dump. =A099% of the threads look very similar= (I'm using 1.1.9 in terms of matching source lines). =A0The thread nam= es are all like this: "WRITE-/10.x.y.z". =A0There are a LOT of du= plicates (in terms of the same IP). =A0Many many many of the threads are tr= ying to talk to IPs that aren't in the cluster (I assume they are the I= P's of dead hosts). =A0The stack trace is basically the same for them a= ll, attached at the bottom. =A0=A0

There is a lot of things I could talk about in terms of= my situation, but what I think might be=A0pertinent=A0to this thread: I hi= t a "tipping point" recently and upgraded a 9 node cluster from A= WS m1.large to m1.xlarge (rolling, one at a time). =A07 of the 9 upgraded f= ine and work great. =A02 of the 9 keep struggling. =A0I've replaced the= m many times now, each time using this process:
And even this morning th= e only two nodes with a high number of threads are those two (yet again). = =A0And at some point they'll OOM.

Seems like there is something about my cluster (caused = by the recent upgrade?) that causes a thread leak on=A0OutboundTcpConnectio= n=A0 =A0But I don't know how to escape from the trap. =A0Any ideas?


--------
=A0 stackTrace = =3D [ {=A0
=A0 =A0 className =3D sun.misc.Unsafe;
=A0 = =A0 fileName =3D Unsafe.java;
=A0 =A0 lineNumber =3D -2;
=A0 =A0 methodName =3D park;
=A0 =A0 nativeMethod =3D true;
=
=A0 =A0}, {=A0
=A0 =A0 className =3D java.util.concurrent.lo= cks.LockSupport;
=A0 =A0 fileName =3D LockSupport.java;
=A0 =A0 lineNumber =3D 158;
=A0 =A0 methodName =3D park;
=A0 =A0 nativeMethod =3D false;=
=A0 =A0}, {=A0
=A0 =A0 className =3D java.util.concurr= ent.locks.AbstractQueuedSynchronizer$ConditionObject;
=A0 =A0 fil= eName =3D AbstractQueuedSynchronizer.java;
=A0 =A0 lineNumber =3D 1987;
=A0 =A0 methodName =3D await;
=A0 =A0 nativeMethod =3D false;
=A0 =A0}, {=A0
=A0 =A0 className =3D java.util.concurrent.LinkedBlockingQueue;
= =A0 =A0 fileName =3D LinkedBlockingQueue.java;
=A0 =A0 lineNumber =3D 399;
=A0 =A0 methodName =3D take;
=A0 =A0 nativeMethod =3D false;
=A0 =A0}, {=A0
= =A0 =A0 className =3D org.apache.cassandra.net.OutboundTcpConnection;
=
=A0 =A0 fileName =3D OutboundTcpConnection.java;
=A0 =A0 lineNumber =3D 104;
=A0 =A0 methodName =3D run;
=A0 =A0 nativeMethod =3D false;
=A0 =A0} ];
-----= -----


<= br>
On Mon, Apr 29, 2013 at 4:31 PM, aaron morton <aaron@thelastpickle.c= om> wrote:
=A0I used JMX to check current number of threads in a production c= assandra machine, and it was ~27,000.
That does not= sound too good.=A0

My first guess would be lots of client connections. What cli= ent are you using, does it do connection pooling ?
See the commen= ts in cassandra.yaml around rpc_server_type, the default uses sync uses one= thread per connection, you may be better with HSHA. But if your app is lea= king connection you should probably deal with that first.=A0

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 30/04/2013, at 3:07 AM, William Oberman <oberman@civicscience.com> wrote:

Hi,
I'm having some issues. =A0I keep getting:
------------<= /div>
ERROR [GossipStage:1] 2013-04-28 07:48:48,876 AbstractCassan= draDaemon.java (line 135) Exception in thread Thread[GossipStage:1,5,main]<= /div>
java.lang.OutOfMemoryError: unable to create new native thread
--------------
after a day or two of runtime. =A0I'v= e checked and my system settings seem acceptable:
memlock=3Dunlim= ited
nofiles=3D100000
nproc=3D122944

I&#= 39;ve messed with heap sizes from 6-12GB (15 physical, m1.xlarge in AWS), a= nd I keep OOM'ing with the above error.

I've found some (what seem to me) to be obscure referenc= es to the stack size interacting with # of threads. =A0If I'm understan= ding it correctly, to reason about Java mem usage I have to think of OS + H= eap as being locked down, and the stack gets the "leftovers" of p= hysical memory and each thread gets a stack.

For me, the system ulimit setting on stack is=A010240k = (no idea if java sees or respects this setting). =A0My -Xss for cassandra i= s the default (I hope, don't remember messing with it) of 180k. =A0I us= ed JMX to check current number of threads in a production cassandra machine= , and it was ~27,000. =A0Is that a normal thread count? =A0Could my OOM be = related to stack + number of threads, or am I overlooking something more si= mple?

will











--bcaec51dd3c7a25eb804dbaeb61b--