incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Faraaz Sareshwala <fsareshw...@quantcast.com>
Subject Large number of pending gossip stage tasks in nodetool tpstats
Date Tue, 06 Aug 2013 19:52:25 GMT
I'm running cassandra-1.2.8 in a cluster with 45 nodes across three racks. All
nodes are well behaved except one. Whenever I start this node, it starts
churning CPU. Running nodetool tpstats, I notice that the number of pending
gossip stage tasks is constantly increasing [1]. When looking at nodetool
gossipinfo, I notice that this node has updated to the latest schema hash, but
that it thinks other nodes in the cluster are on the older version. I've tried
to drain, decommission, wipe node data, bootstrap, and repair the node. However,
the node just started doing the same thing again.

Has anyone run into this issue before? Can anyone provide any insight into why
this node is the only one in the cluster having problems? Are there any easy
fixes?

Thank you,
Faraaz

[1] $ /cassandra/bin/nodetool tpstats
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
ReadStage                         0         0              8         0                 0
RequestResponseStage              0         0          49198         0                 0
MutationStage                     0         0         224286         0                 0
ReadRepairStage                   0         0              0         0                 0
ReplicateOnWriteStage             0         0              0         0                 0
GossipStage                       1      2213             18         0                 0
AntiEntropyStage                  0         0              0         0                 0
MigrationStage                    0         0             72         0                 0
MemtablePostFlusher               0         0            102         0                 0
FlushWriter                       0         0             99         0                 0
MiscStage                         0         0              0         0                 0
commitlog_archiver                0         0              0         0                 0
InternalResponseStage             0         0             19         0                 0
HintedHandoff                     0         0              2         0                 0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                  0
BINARY                       0
READ                         0
MUTATION                     0
_TRACE                       0
REQUEST_RESPONSE             0

Mime
View raw message