cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Roth <benjamin.r...@jaumo.com>
Subject Re: CASSANDRA-12888: Streaming and MVs
Date Wed, 07 Dec 2016 08:01:04 GMT
Hi Paolo,

First of all thanks for your review!

I had the same concerns as you but I thought it is beeing handled correctly
(which does in some situations) but I found one that creates the
inconsistencies you mentioned. That is kind of split brain syndrom, when
multiple nodes fail between repairs. See here: https://cl.ly/3t0X1c0q1L1h.

I am not happy about it but I support your decision. We should then add
another dtest to test this scenario as existing dtests don't.

Some issues unfortunately remain:
- 12888 is not resolved
- MV repairs may be still f**** slow. Imagine an inconsistency of a single
cell (also may be due to a validation race condition, see CASSANDRA-12991)
on a big partition. I had issues with reaper and a 30min timeout leading to
1000+ (yes!) consecutive repairs of a single subrange because it always
timed out and I recognized very late. When I deployed 12888 on my system,
this remaining subrange was repaired in a snap
- I guess rebuild works the same as repair and has to go through the write
path, right?

=> The MV repair may induce so much overhead that it is maybe cheaper to
kill and replace a inconsistent node than to repair it. But that may
introduce inconsistencies again. All in all it is not perfect. All this
does not really un-frustrate me a 100%.

Do you have any more thoughts?

Unfortunately I have very little time these days as my second child was
born on monday. So thanks for your support so far. Maybe I have some ideas
on this issues during the next days and I will work on that ticket probably
next week to come to a solution that is at least deployable. I'd also
appreciate your opinion on CASSANDRA-12991.

2016-12-07 2:53 GMT+01:00 Paulo Motta <pauloricardomg@gmail.com>:

> Hello Benjamin,
>
> Thanks for your effort on this investigation! For bootstraps and range
> transfers, I think we can indeed simplify and stream base tables and MVs as
> ordinary tables, unless there is some caveat I'm missing (I didn't find any
> special case for bootstrap/range transfers on CASSANDRA-6477 or in the MV
> design doc, please correct me if I'm wrong).
>
> Regarding repair of base tables, applying mutations via the write path is a
> matter of correctness, given that the base table updates needs to
> potentially remove previously referenced keys in the views, so repairing
> only the base table may leave unreferenced keys in the views, breaking the
> MV contract. Furthermore, these unreferenced keys may be propagated to
> other replicas and never removed if you repair only the view. If you don't
> do overwrites in the base table, this is probably not a problem but the DB
> cannot ensure this (at least not before CASSANDRA-9779). Furthermore, as
> you already noticed repairing only the base table is probably faster so I
> don't see a reason to repair the base and MVs separately since this is
> potentially more costly. I believe your frustration is mostly due to the
> bug described on CASSANDRA-12905, but after that and CASSANDRA-12888 are
> fixed repair on base table should work just fine.
>
> Based on this, I propose:
> - Fix CASSANDRA-12905 with your original patch that retries acquiring the
> MV lock instead of throwing WriteTimeoutException during streaming, since
> this is blocking 3.10.
> - Fix CASSANDRA-12888 by doing sstable-based streaming for base tables
> while still applying MV updates in the paired replicas.
> - Create new ticket to use ordinary streaming for non-repair MV stream
> sessions and keep current behavior for MV streaming originating from
> repair.
> - Create new ticket to include only the base tables and not MVs in
> keyspace-level repair, since repairing the base already repairs the views
> to avoid people shooting themselves in the foot.
>
> Please let me know what do you think. Any suggestions or feedback is
> appreciated.
>
> Cheers,
>
> Paulo
>
> 2016-12-02 8:27 GMT-02:00 Benjamin Roth <benjamin.roth@jaumo.com>:
>
> > As I haven't received a single reply on that, I went over to implement
> and
> > test it on my own with our production cluster. I had a real pain with
> > bringing up a new node, so I had to move on.
> >
> > Result:
> > Works like a charm. I ran many dtests that relate in any way with
> storage,
> > stream, bootstrap, ... with good results.
> > The bootstrap finished in under 5:30h, not a single error log during
> > bootstrap. Also afterwards, repairs run smooth, cluster seems to operate
> > quite well.
> >
> > I still need:
> >
> >    - Reviews (see 12888, 12905, 12984)
> >    - Some opinion if I did the CDC case right. IMHO CDC is not required
> on
> >    bootstrap and we don't need to send the mutations through the write
> path
> >    just to write the commit log. This will also break incremental
> repairs.
> >    Instead for CDC the sstables are streamed like normal but mutations
> are
> >    written to commitlog additionally. The worst I see is that the node
> > crashes
> >    and the commitlogs for that repair streams are replayed leading to
> >    duplicate writes, which is not really crucial and not a regular case.
> > Any
> >    better ideas?
> >    - Docs have to be updated (12985) if patch is accepted
> >
> > I really appreciate ANY feedback. IMHO the impact of that fixes is
> immense
> > and maybe will be a huge step to get MVs production ready.
> >
> > Thank you very much,
> > Benjamin
> >
> >
> > ---------- Forwarded message ----------
> > From: Benjamin Roth <benjamin.roth@jaumo.com>
> > Date: 2016-11-29 17:04 GMT+01:00
> > Subject: Streaming and MVs
> > To: dev@cassandra.apache.org
> >
> >
> > I don't know where else to discuss this issue, so I post it here.
> >
> > I am trying to get CS to run stable with MVs since the beginning of july.
> > Normal reads + write do work as expected but when it comes to repairs or
> > bootstrapping it still feels far far away from what I would call fast and
> > stable. The other day I just wanted to bootstrap a new node. I tried it 2
> > times.
> > First time the bootstrap failed due to WTEs. I fixed this issue by not
> > timing out in streams but then it turned out that the bootstrap (load
> > roughly 250-300 GB) didn't even finish in 24h. What if I really had a
> > problem and had to get up some nodes fast? No way!
> >
> > I think the root cause of it all is the way streams are handled on tables
> > with MVs.
> > Sending them to the regular write path implies so many bottlenecks and
> > sometimes also redundant writes. Let me explain:
> >
> > 1. Bootstrap
> > During a bootstrap, all ranges from all KS and all CFs that will belong
> to
> > the new node will be streamed. MVs are treated like all other CFs and all
> > ranges that will move to the new node will also be streamed during
> > bootstrap.
> > Sending streams of the base tables through the write path will have the
> > following negative impacts:
> >
> >    - Writes are sent to the commit log. Not necessary. When node is
> stopped
> >    during bootstrap, bootstrap will simply start over. No need to recover
> > from
> >    commit logs. Non-MV tables won't have a CL anyway
> >    - MV mutations will not be applied instantly but send to the batch
> log.
> >    This is of course necessary during the range movement (if PK of MV
> > differs
> >    from base table) but what happens: The batchlog will be completely
> > flooded.
> >    This leads to ridiculously large batchlogs (I observerd BLs with 60GB
> >    size), zillions of compactions and quadrillions of tombstones. This
> is a
> >    pure resource killer, especially because BL uses a CF as a queue.
> >    - Applying every mutation separately causes read-before-writes during
> MV
> >    mutation. This is of course an order of magnitude slower than simply
> >    streaming down an SSTable. This effect becomes even worse while
> > bootstrap
> >    progresses and creates more and more (uncompacted) SSTables. Many of
> > them
> >    wont ever be compacted because the batchlog eats all the resources
> >    available for compaction
> >    - Streaming down the MV tables AND applying the mutations of the
> >    basetables leads to redundant writes. Redundant writes are local if PK
> > of
> >    the MV == PK of the base table and - even worse - remote if not.
> Remote
> > MV
> >    updates will impact nodes that aren't even part of the bootstrap.
> >    - CDC should also not be necessary during bootstrap, should it? TBD
> >
> > 2. Repair
> > Negative impact is similar to bootstrap but, ...
> >
> >    - Sending repairs through write path will not mark the streamed tables
> >    as repaired. See CASSANDRA-12888. Doing NOT so will instantly solve
> that
> >    issue. Much simpler with any other solution
> >    - It will change the "repair design" a bit. Repairing a base table
> will
> >    not automatically repair the MV. But is this bad at all? To be honest
> > as a
> >    newbie it was very hard for me to understand what I had to do to be
> sure
> >    that everything is repaired correctly. Recently I was told NOT to
> > repair MV
> >    CFs but only to repair the base tables. This means one cannot just
> call
> >    "nodetool repair $keyspace" - this is complicated, not transparent and
> > it
> >    sucks. I changed the behaviour in my own branch and let run the dtests
> > for
> >    MVs. 2 tests failed:
> >       - base_replica_repair_test of course failes due to the design
> change
> >       - really_complex_repair_test fails because it intentionally times
> out
> >       the batch log. IMHO this is a bearable situation. It is comparable
> to
> >       resurrected tombstones when running a repair after GCGS expired.
> You
> > also
> >       would not expect this to be magically fixed. gcgs default is 10
> > days and I
> >       can expect that anybody also repairs its MVs during that period,
> not
> > only
> >       the base table
> >
> > 3. Rebuild
> > Same like bootstrap, isn't it?
> >
> > Did I forget any cases?
> > What do you think?
> >
> > --
> > Benjamin Roth
> > Prokurist
> >
> > Jaumo GmbH · www.jaumo.com
> > Wehrstraße 46 · 73035 Göppingen · Germany
> > Phone +49 7161 304880-6 <07161%203048806> · Fax +49 7161 304880-1
> > <07161%203048801>
> > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
> >
> >
> >
> > --
> > Benjamin Roth
> > Prokurist
> >
> > Jaumo GmbH · www.jaumo.com
> > Wehrstraße 46 · 73035 Göppingen · Germany
> > Phone +49 7161 304880-6 · Fax +49 7161 304880-1
> > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
> >
>



-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message