cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Yang"...@iyyang.com>
Subject Re: Why is mutation stage increasing ??
Date Wed, 05 Oct 2011 08:52:34 GMT
Well what client are you using? And can you give a hint to your node hardware?

從我的 BlackBerry® 無線裝置

-----Original Message-----
From: Philippe <watcherfr@gmail.com>
Date: Wed, 5 Oct 2011 10:33:21 
To: user<user@cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Subject: Why is mutation stage increasing ??

Hello,
I have my 3-node, RF=3 cluster acting strangely. Can someone shed a light as
to what is going on ?
It was stuck for a couple of hours (all clients TimedOut). nodetool tpstats
showed huge increasing MutationStages (in the hundreds of thousands).
I restarted one node and it took a while to reply GB of commitlog. I've
shutdown all clients that write to the cluster and it's just weird

All nodes are still showing huge MutationStages including the new one and
it's either increasing or stable. The pending count is stuck at 32.
Compactionstats shows no compaction on 2 nodes and dozens of Scrub
compactions (all at 100%) on the 3rd one. This is a scrub I did last week
when I encountered assertion errors.
Netstats shows no streams being exchanged at any node but each on is
expecting a few Responses.

Any ideas ?
Thanks

For example (increased to 567062 while I was writing this email)
Pool Name                    Active   Pending      Completed   Blocked  All
time blocked
ReadStage                         0         0    18372664517
0                 0
RequestResponseStage              0         0    10731370183
0                 0
MutationStage                    32    565879      295492216
0                 0
ReadRepairStage                   0         0          23654
0                 0
ReplicateOnWriteStage             0         0        7733659
0                 0
GossipStage                       0         0        3502922
0                 0
AntiEntropyStage                  0         0           1631
0                 0
MigrationStage                    0         0              0
0                 0
MemtablePostFlusher               0         0           5716
0                 0
StreamStage                       0         0             10
0                 0
FlushWriter                       0         0           5714
0               499
FILEUTILS-DELETE-POOL             0         0            773
0                 0
MiscStage                         0         0           1266
0                 0
FlushSorter                       0         0              0
0                 0
AntiEntropySessions               0         0             18
0                 0
InternalResponseStage             0         0              0
0                 0
HintedHandoff                     0         0           1798
0                 0


Mode: Normal
Not sending any streams.
Not receiving any streams.
Pool Name                    Active   Pending      Completed
Commands                        n/a         0     1223769753
Responses                       n/a         4     1627481305

Mime
View raw message