incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Freeman <>
Subject Re: heavy insert load overloads CPUs, with MutationStage pending
Date Tue, 10 Sep 2013 17:40:23 GMT

On 09/10/2013 11:17 AM, Robert Coli wrote:
> On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman < 
> <>> wrote:
>     On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for
>     commitlog and data
> On SSD, you don't need to separate commitlog and data. You only win 
> from this separation if you have a head to not-move between appends to 
> the commit log. You will get better IO from a strip with an additional 
> SSD.
Right, actually both partitions are on the same SSD.   Assuming you 
meant "stripe", would that really make a difference
>         Pool Name                    Active   Pending  Completed  
>         Blocked  All time blocked
>         MutationStage                     1         9 290394         0
>                         0
>         FlushWriter                       1         2   20         0  
>                       0
>     I can't seem find information about the real meaning of
>     MutationStage, is this just normal for lots of inserts?
> The mutation stage is the stage in which mutations to rows in 
> memtables ("writes") occur.
> The FlushWriter stage is the stage that turns memtables into SSTables 
> by flushing them.
> However, 9 pending mutations is a very small number. For reference on 
> an overloaded cluster which was being written to death I recently 
> saw.... 1216434 pending MutationStage. What problem other than "high 
> CPU load" are you experiencing? 2 Pending FlushWriters is slightly 
> suggestive of some sort of bound related to flushing..
So the basic problem is that write performance is lower than I 
expected.  I can't get sustained writing of 5000 ~1024-byte records / 
sec at RF=2 on a good 3-node cluster, and my only guess is that's 
because of the heavy CPU loads on the server (loads over 10 on 4-CPU 
systems).  I've tried both a single client writing 5000 rows/second and 
2 clients (on separate boxes) writing 2500 rows/second, and in both 
cases the server(s) doesn't respond quickly enough to maintain that 
rate.  It keeps up ok with 2000 or 3000 rows per second (and has lower 
server loads).

View raw message