cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: memtablePostFlusher blocking writes?
Date Wed, 27 Apr 2011 18:09:33 GMT
MPF is indeed pretty lightweight, but since its job is to mark the
commitlog replay position after a flush -- which has to be done in
flush order to preserve correctness in failure scenarios -- you'll see
the pending op count go up when you have multiple flushes happening.
This is expected.

Your real problem is the 17000 pending mutations, the 22 active +
pending flushes, and probably compaction activity as well.

(Also, if each of those pending mutations is 10,000 columns, you may
be causing yourself memory pressure as well.)

On Wed, Apr 27, 2011 at 11:01 AM, Terje Marthinussen
<tmarthinussen@gmail.com> wrote:
> 0.8 trunk:
>
> When playing back a fairly large chunk of hints, things basically locks up
> under load.
> The hints are never processed successfully. Lots of Mutations dropped.
>
> One thing is that maybe the default 10k columns per send with 50ms delays is
> a bit on the aggressive side (10k*20 =200.000 columns in a second?), the
> other thing is that it seems like the whole memtable flushing locks up.
>
> I tried to increase number of memtable flushers and queue a bit (8
> concurrent flushers) to make things work, but no luck.
>
> Pool Name                    Active   Pending      Completed
> ReadStage                         0         0             
1
> RequestResponseStage              0         0        2236304
> MutationStage                   100     17564        4011533
> ReadRepairStage                   0         0             
0
> ReplicateOnWriteStage             0         0             
0
> GossipStage                       0         0          
2281
> AntiEntropyStage                  0         0             
0
> MigrationStage                    0         0             
0
> MemtablePostFlusher               1        13            
50
> StreamStage                       0         0             
0
> FlushWriter                       8        14            
73
> MiscStage                         0         0             
0
> FlushSorter                       0         0             
0
> InternalResponseStage             0         0             
0
> HintedHandoff                     1         8             
3
>
> A quick source code scan makes me believe that the MemtablePostFlusher
> should not normally use a lot of time, but it seem like it does so here.
> What may cause this?
>
> Terje
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Mime
View raw message