cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: repair question
Date Tue, 24 May 2011 20:32:51 GMT
> And the more spread out the out of sync is, the worse it will be. Though in
> general we can expect that to not be too spread out. For the same reason than
> why caches work.

(Speaking generally now, not addressing the OP's issue)

I'm not sure I buy that. Unless your data is such that hotness is
expected to be related somehow to the order of the data, even with a
workload quite susceptible to caching you very quickly spread requests
out lots of merkle ring segments.

For example, suppose you have a CF with lots of smallish rows - say
250 byte per row on average. Given a 50 GB load and 2^15 for the range
(let's assume 0.8 improvements), each chunk/merkle segment will be 1.6
mb in size. You don't need to hit a lot of unique keys in order to
make the next repair transfer most content. 2^15 is only 32k; if a
node is e.g. overloaded or some other event causes more than just
random sporadic message dropping for any non-trivial period, it is not
difficult at all to hit >> 32k unique keys being written to for pretty
plausible workloads.

It's somewhat of a concern particularly because of the original reason
for dropped keys is overload, and you have lots of data and are
bottlenecking on I/O, the "over-repair" will then cause further data
bloat (even if temporarily), causing more dropped messages, causing
the next repair to do the same, etc.
> So maybe it is worth improving the granularity of the tree. It is true that
> having a fixed granularity as we do is probably a bit naive. In any case, it
> will still be a trade-off between the precision of the tree and it's size
> (which impact memory used and we have to transfer it (though the transfer part
> could be done by tree level -- right now the whole tree is always transfered
> which is not so optimal)).

Longer term I'm thinking something more incremental and less "bulky"
is desired, and such a solution might have the side-effect of avoiding
the granularity problem.

For example, consider a repair process where the originating node
iterates over its own range in a compaction-like fashion, calculating
small merkle trees as required. The node could ask neighbors to
deliver relevant data in a fashion more similar to hinted hand-off or
read-repair. You'd now have an incremental process that behaves more
like normal writes in terms of the impact on data size, compaction etc
instead of a sudden spike in sstable sizes and large long-running
compaction/streaming jobs.

Further the process should be fairly insensitive to issues like total
data size, the average amount of rows out of synch, etc. It could grab
one chunk at a time, with a "chunk" being some fixed reasonable size
like say 1 GB spread out over appropriate sstables (i..e, take
whatever segment of the ring corresponds to 1 GB). There would be no
need to keep long-term refs to obsolete sstables during this process
since the iteration would be entirely in token order so both the
receiver and the sender in the synchronization process should see a
minimal impact in terms of sstable retention, backed up compactions,
etc. By doing chunks however, you avoid the cost of traversing the
data in a seek bound fashion, since each chunk of reasonable size
would have compaction-like performance characteristics.

Hmmm, I'm starting to like this idea more and more the more I think of it ;)

/ Peter Schuller

View raw message