cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Why is mutation stage increasing ??
Date Wed, 05 Oct 2011 09:05:32 GMT
Lots of hinted handoff can give you mutations…

> HintedHandoff                     0         0           1798         0              
  0


1798 is somewhat high. This is the HH tasks on this node though, can you see HH running on
other nodes in the cluster? What has been happening on this node ? 

HH is throttled to avoid this sort of thing, what version are you on ? 

Also looks like the disk IO could not keep up with the flushing….

FlushWriter                       0         0           5714         0               499
 
You need to provide some more info on what was happening to nodes before hand. And check the
logs on all machines for errors etc. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 5/10/2011, at 9:52 PM, Yi Yang wrote:

> Well what client are you using? And can you give a hint to your node hardware?
> 從我的 BlackBerry® 無線裝置
> 
> From: Philippe <watcherfr@gmail.com>
> Date: Wed, 5 Oct 2011 10:33:21 +0200
> To: user<user@cassandra.apache.org>
> ReplyTo: user@cassandra.apache.org
> Subject: Why is mutation stage increasing ??
> 
> Hello,
> I have my 3-node, RF=3 cluster acting strangely. Can someone shed a light as to what
is going on ?
> It was stuck for a couple of hours (all clients TimedOut). nodetool tpstats showed huge
increasing MutationStages (in the hundreds of thousands).
> I restarted one node and it took a while to reply GB of commitlog. I've shutdown all
clients that write to the cluster and it's just weird
> 
> All nodes are still showing huge MutationStages including the new one and it's either
increasing or stable. The pending count is stuck at 32.
> Compactionstats shows no compaction on 2 nodes and dozens of Scrub compactions (all at
100%) on the 3rd one. This is a scrub I did last week when I encountered assertion errors.
> Netstats shows no streams being exchanged at any node but each on is expecting a few
Responses.
> 
> Any ideas ?
> Thanks
> 
> For example (increased to 567062 while I was writing this email)
> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> ReadStage                         0         0    18372664517         0              
  0
> RequestResponseStage              0         0    10731370183         0              
  0
> MutationStage                    32    565879      295492216         0              
  0
> ReadRepairStage                   0         0          23654         0              
  0
> ReplicateOnWriteStage             0         0        7733659         0              
  0
> GossipStage                       0         0        3502922         0              
  0
> AntiEntropyStage                  0         0           1631         0              
  0
> MigrationStage                    0         0              0         0              
  0
> MemtablePostFlusher               0         0           5716         0              
  0
> StreamStage                       0         0             10         0              
  0
> FlushWriter                       0         0           5714         0              
499
> FILEUTILS-DELETE-POOL             0         0            773         0              
  0
> MiscStage                         0         0           1266         0              
  0
> FlushSorter                       0         0              0         0              
  0
> AntiEntropySessions               0         0             18         0              
  0
> InternalResponseStage             0         0              0         0              
  0
> HintedHandoff                     0         0           1798         0              
  0
> 
> 
> Mode: Normal
> Not sending any streams.
> Not receiving any streams.
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0     1223769753
> Responses                       n/a         4     1627481305
> 


Mime
View raw message