incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sankalp kohli <kohlisank...@gmail.com>
Subject Re: heavy insert load overloads CPUs, with MutationStage pending
Date Tue, 10 Sep 2013 19:59:54 GMT
What have you set these to?
# commitlog_sync may be either "periodic" or "batch."
# When in batch mode, Cassandra won't ack writes until the commit log
# has been fsynced to disk.  It will wait up to
# commitlog_sync_batch_window_in_ms milliseconds for other writes, before
# performing the sync.
#
# commitlog_sync: batch
# commitlog_sync_batch_window_in_ms: 50
#
# the other option is "periodic" where writes may be acked immediately
# and the CommitLog is simply synced every commitlog_sync_period_in_ms
# milliseconds.
commitlog_sync: periodic
commitlog_sync_period_in_ms: 1000


On Tue, Sep 10, 2013 at 10:42 AM, Nate McCall <nate@thelastpickle.com>wrote:

> With SSDs, you can turn up memtable_flush_writers - try 3 initially (1 by
> default) and see what happens. However, given that there are no entries in
> 'All time blocked' for such, they may be something else.
>
> How are you inserting the data?
>
>
> On Tue, Sep 10, 2013 at 12:40 PM, Keith Freeman <8forty@gmail.com> wrote:
>
>>
>> On 09/10/2013 11:17 AM, Robert Coli wrote:
>>
>> On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman <8forty@gmail.com> wrote:
>>
>>> On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog
>>> and data
>>
>>
>>  On SSD, you don't need to separate commitlog and data. You only win
>> from this separation if you have a head to not-move between appends to the
>> commit log. You will get better IO from a strip with an additional SSD.
>>
>> Right, actually both partitions are on the same SSD.   Assuming you meant
>> "stripe", would that really make a difference
>>
>>
>>
>>>  Pool Name                    Active   Pending      Completed   Blocked
>>>>  All time blocked
>>>> MutationStage                     1         9         290394         0
>>>>                 0
>>>> FlushWriter                       1         2             20         0
>>>>                 0
>>>>
>>>
>>
>>>  I can't seem find information about the real meaning of MutationStage,
>>> is this just normal for lots of inserts?
>>>
>>
>>  The mutation stage is the stage in which mutations to rows in memtables
>> ("writes") occur.
>>
>>  The FlushWriter stage is the stage that turns memtables into SSTables
>> by flushing them.
>>
>>  However, 9 pending mutations is a very small number. For reference on
>> an overloaded cluster which was being written to death I recently saw....
>> 1216434 pending MutationStage. What problem other than "high CPU load" are
>> you experiencing? 2 Pending FlushWriters is slightly suggestive of some
>> sort of bound related to flushing..
>>
>> So the basic problem is that write performance is lower than I expected.
>> I can't get sustained writing of 5000 ~1024-byte records / sec at RF=2 on a
>> good 3-node cluster, and my only guess is that's because of the heavy CPU
>> loads on the server (loads over 10 on 4-CPU systems).  I've tried both a
>> single client writing 5000 rows/second and 2 clients (on separate boxes)
>> writing 2500 rows/second, and in both cases the server(s) doesn't respond
>> quickly enough to maintain that rate.  It keeps up ok with 2000 or 3000
>> rows per second (and has lower server loads).
>>
>>
>>
>

Mime
View raw message