incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Freeman <8fo...@gmail.com>
Subject Re: heavy insert load overloads CPUs, with MutationStage pending
Date Wed, 11 Sep 2013 16:17:39 GMT
I have the defaults as shown in your response.

On 09/10/2013 01:59 PM, sankalp kohli wrote:
> What have you set these to?
> # commitlog_sync may be either "periodic" or "batch."
> # When in batch mode, Cassandra won't ack writes until the commit log
> # has been fsynced to disk.  It will wait up to
> # commitlog_sync_batch_window_in_ms milliseconds for other writes, before
> # performing the sync.
> #
> # commitlog_sync: batch
> # commitlog_sync_batch_window_in_ms: 50
> #
> # the other option is "periodic" where writes may be acked immediately
> # and the CommitLog is simply synced every commitlog_sync_period_in_ms
> # milliseconds.
> commitlog_sync: periodic
> commitlog_sync_period_in_ms: 1000
>
>
> On Tue, Sep 10, 2013 at 10:42 AM, Nate McCall <nate@thelastpickle.com 
> <mailto:nate@thelastpickle.com>> wrote:
>
>     With SSDs, you can turn up memtable_flush_writers - try 3
>     initially (1 by default) and see what happens. However, given that
>     there are no entries in 'All time blocked' for such, they may be
>     something else.
>
>     How are you inserting the data?
>
>
>     On Tue, Sep 10, 2013 at 12:40 PM, Keith Freeman <8forty@gmail.com
>     <mailto:8forty@gmail.com>> wrote:
>
>
>         On 09/10/2013 11:17 AM, Robert Coli wrote:
>>         On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman
>>         <8forty@gmail.com <mailto:8forty@gmail.com>> wrote:
>>
>>             On my 3-node cluster (v1.2.8) with 4-cores each and SSDs
>>             for commitlog and data
>>
>>
>>         On SSD, you don't need to separate commitlog and data. You
>>         only win from this separation if you have a head to not-move
>>         between appends to the commit log. You will get better IO
>>         from a strip with an additional SSD.
>         Right, actually both partitions are on the same SSD.  
>         Assuming you meant "stripe", would that really make a difference
>
>>                 Pool Name  Active   Pending      Completed Blocked
>>                  All time blocked
>>                 MutationStage 1         9         290394 0          
>>                       0
>>                 FlushWriter 1         2             20 0            
>>                     0
>>
>>             I can't seem find information about the real meaning of
>>             MutationStage, is this just normal for lots of inserts?
>>
>>
>>         The mutation stage is the stage in which mutations to rows in
>>         memtables ("writes") occur.
>>
>>         The FlushWriter stage is the stage that turns memtables into
>>         SSTables by flushing them.
>>
>>         However, 9 pending mutations is a very small number. For
>>         reference on an overloaded cluster which was being written to
>>         death I recently saw.... 1216434 pending MutationStage. What
>>         problem other than "high CPU load" are you experiencing? 2
>>         Pending FlushWriters is slightly suggestive of some sort of
>>         bound related to flushing..
>         So the basic problem is that write performance is lower than I
>         expected.  I can't get sustained writing of 5000 ~1024-byte
>         records / sec at RF=2 on a good 3-node cluster, and my only
>         guess is that's because of the heavy CPU loads on the server
>         (loads over 10 on 4-CPU systems).  I've tried both a single
>         client writing 5000 rows/second and 2 clients (on separate
>         boxes) writing 2500 rows/second, and in both cases the
>         server(s) doesn't respond quickly enough to maintain that
>         rate.  It keeps up ok with 2000 or 3000 rows per second (and
>         has lower server loads).
>
>
>
>


Mime
View raw message