incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Terje Marthinussen <tmarthinus...@gmail.com>
Subject Re: memtablePostFlusher blocking writes?
Date Wed, 27 Apr 2011 22:23:32 GMT
It is a good question what is the problem here.
I dont think it is the pending mutations and flushes, the real problem is
what causes them, and it is not me!

There was maybe a misleading comment in my original mail.
It is not the hinted handoffs sent from this node that is the problem, but
the 1.6 million hints for this node that is being sent to it from another
(neighbour) node.

A more compact view what happens when the other node starts playing back the
hints. There is 2 seconds between the lines.
MutationStage                   100       615        2008362
MutationStage                   100      1202        2054437
MutationStage                   100      1394        2104971
MutationStage                   100      9318        2142964
MutationStage                   100     46392        2142964
MutationStage                   100     83188        2142964
MutationStage                   100    113156        2142964
MutationStage                   100    149063        2142964
MutationStage                   100    187514        2142964
MutationStage                   100    226238        2142964
MutationStage                   100    146267        2264194
MutationStage                   100    141232        2314345
MutationStage                   100    129730        2366987
MutationStage                   100    128580        2412014
MutationStage                   100    124101        2460093
MutationStage                   100    119032        2509960
MutationStage                   100    126888        2538537
MutationStage                   100    163049        2538537
MutationStage                   100    197243        2538537
MutationStage                   100    231564        2538537
MutationStage                   100     95212        2675457
MutationStage                   100     43066        2727606
MutationStage                    26       127        2779756
MutationStage                   100      1115        2822694
MutationStage                    22        22        2873449

I have two issues here.
- The massive amount of mutation caused by the hints playback
- The long periods where there are no increase in completed mutations.

Yes, there are other things going here when this happens, the system is
actually fairly busy, but has no problem handling the other traffic. The
RoundRobin Scheduler is also activated with 100 as throttle limit on a 12
node cluster, so there is no way at all that you should see 30k mutations
increasing on the pending side from the feeding side. This seems completely
triggered by the hints coming in.

I tried to reduce PAGE_SIZE to 5000 and set the throttle limit all the way
up to 5000ms, still it does not really seem to help much. It just take
longer time and it seems to be memtable flushing that blocks things.

Terje



On Thu, Apr 28, 2011 at 3:09 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> MPF is indeed pretty lightweight, but since its job is to mark the
> commitlog replay position after a flush -- which has to be done in
> flush order to preserve correctness in failure scenarios -- you'll see
> the pending op count go up when you have multiple flushes happening.
> This is expected.
>
> Your real problem is the 17000 pending mutations, the 22 active +
> pending flushes, and probably compaction activity as well.
>
> (Also, if each of those pending mutations is 10,000 columns, you may
> be causing yourself memory pressure as well.)
>
> On Wed, Apr 27, 2011 at 11:01 AM, Terje Marthinussen
> <tmarthinussen@gmail.com> wrote:
> > 0.8 trunk:
> >
> > When playing back a fairly large chunk of hints, things basically locks
> up
> > under load.
> > The hints are never processed successfully. Lots of Mutations dropped.
> >
> > One thing is that maybe the default 10k columns per send with 50ms delays
> is
> > a bit on the aggressive side (10k*20 =200.000 columns in a second?), the
> > other thing is that it seems like the whole memtable flushing locks up.
> >
> > I tried to increase number of memtable flushers and queue a bit (8
> > concurrent flushers) to make things work, but no luck.
> >
> > Pool Name                    Active   Pending      Completed
> > ReadStage                         0         0              1
> > RequestResponseStage              0         0        2236304
> > MutationStage                   100     17564        4011533
> > ReadRepairStage                   0         0              0
> > ReplicateOnWriteStage             0         0              0
> > GossipStage                       0         0           2281
> > AntiEntropyStage                  0         0              0
> > MigrationStage                    0         0              0
> > MemtablePostFlusher               1        13             50
> > StreamStage                       0         0              0
> > FlushWriter                       8        14             73
> > MiscStage                         0         0              0
> > FlushSorter                       0         0              0
> > InternalResponseStage             0         0              0
> > HintedHandoff                     1         8              3
> >
> > A quick source code scan makes me believe that the MemtablePostFlusher
> > should not normally use a lot of time, but it seem like it does so here.
> > What may cause this?
> >
> > Terje
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Mime
View raw message