Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Tue, 22 Jan 2013 19:08:13 +0000 (UTC)
From: "Brandon Williams (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12628423.1358625179379.174327.1358881693518@arcas>
In-Reply-To: <JIRA.12628423.1358625179379@arcas>
References: <JIRA.12628423.1358625179379@arcas>
Subject: [jira] [Commented] (CASSANDRA-5175) Unbounded (?) thread growth
 connecting to an removed node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559896#comment-13559896 ] 

Brandon Williams commented on CASSANDRA-5175:
---------------------------------------------

Hard to know for certain if this is truly the cause, but it sounds plausible. +1
                
> Unbounded (?) thread growth connecting to an removed node
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-5175
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5175
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.8
>         Environment: EC2, JDK 7u9, Ubuntu 12.04.1 LTS
>            Reporter: Janne Jalkanen
>            Assignee: Vijay
>            Priority: Minor
>             Fix For: 1.1.10
>
>         Attachments: 0001-CASSANDRA-5175.patch
>
>
> The following lines started repeating every minute in the log file
> {noformat}
>  INFO [GossipStage:1] 2013-01-19 19:35:43,929 Gossiper.java (line 831) InetAddress /10.238.x.y is now dead.
>  INFO [GossipStage:1] 2013-01-19 19:35:43,930 StorageService.java (line 1291) Removing token 170141183460469231731687303715884105718 for /10.238.x.y
> {noformat}
> Also, I got about 3000 threads which all look like this:
> {noformat}
> Name: WRITE-/10.238.x.y
> State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bb65c0f
> Total blocked: 0  Total waited: 3
> Stack trace: 
>  sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:104)
> {noformat}
> A new thread seems to be created every minute, and they never go away.
> The endpoint in question had been a part of the cluster weeks ago, and the node exhibiting the thread growth was added yesterday.
> Anyway, assassinating the endpoint in question stopped thread growth (but kept the existing threads running), so this isn't a huge issue.  But I don't think the thread count is supposed to be increasing like this...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira