incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Freeman <8fo...@gmail.com>
Subject heavy insert load overloads CPUs, with MutationStage pending
Date Tue, 10 Sep 2013 14:55:56 GMT
On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog 
and data, I get high CPU loads during a heavy-ish wide-row insert load 
into a single CF (5000 1k inserts/sec), e.g. uptime load avg for last 
minute 18/11/10.  Checking tpstats, I see MutationStage pending on all 
the nodes, e.g.:

> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> ReadStage                         0         0            144         0              
  0
> RequestResponseStage              0         0         243529         0              
  0
> MutationStage                     1         9         290394         0              
  0
> ReadRepairStage                   0         0              0         0              
  0
> ReplicateOnWriteStage             0         0              0         0              
  0
> GossipStage                       0         0           1014         0              
  0
> AntiEntropyStage                  0         0              0         0              
  0
> MigrationStage                    0         0             13         0              
  0
> MemtablePostFlusher               1         2             35         0              
  0
> FlushWriter                       1         2             20         0              
  0
> MiscStage                         0         0              1         0              
  0
> commitlog_archiver                0         0              0         0              
  0

I can't seem find information about the real meaning of MutationStage, 
is this just normal for lots of inserts?

Also, switching from spinning disks to SSDs didn't seem to significantly 
improve insert performance, so it seems clear my use-case it totally 
CPU-bound.  Cassandra docs say "Insert-heavy workloads are CPU-bound in 
Cassandra before becoming memory-bound.", so I guess that's what I'm 
seeing, but there's no explanation. So I'm wonder what's overloading my 
CPUs, and is there anything I can do about it short of adding more nodes?

.


Mime
View raw message