cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Janne Jalkanen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-5175) Unbounded (?) thread growth connecting to an removed node
Date Sat, 19 Jan 2013 19:54:12 GMT
Janne Jalkanen created CASSANDRA-5175:
-----------------------------------------

             Summary: Unbounded (?) thread growth connecting to an removed node
                 Key: CASSANDRA-5175
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5175
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.1.8
         Environment: EC2, JDK 7u9, Ubuntu 12.04.1 LTS
            Reporter: Janne Jalkanen
            Priority: Minor


The following lines started repeating every minute in the log file

{noformat}
 INFO [GossipStage:1] 2013-01-19 19:35:43,929 Gossiper.java (line 831) InetAddress /10.238.x.y
is now dead.
 INFO [GossipStage:1] 2013-01-19 19:35:43,930 StorageService.java (line 1291) Removing token
170141183460469231731687303715884105718 for /10.238.x.y
{noformat}

Also, I got about 3000 threads which all look like this:

{noformat}
Name: WRITE-/10.238.x.y
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bb65c0f
Total blocked: 0  Total waited: 3

Stack trace: 
 sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:104)
{noformat}

A new thread seems to be created every minute, and they never go away.

The endpoint in question had been a part of the cluster weeks ago, and the node exhibiting
the thread growth was added yesterday.

Anyway, assassinating the endpoint in question stopped thread growth (but kept the existing
threads running), so this isn't a huge issue.  But I don't think the thread count is supposed
to be increasing like this...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message