cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Brown (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-7203) Flush (and Compact) High Traffic Partitions Separately
Date Tue, 02 Dec 2014 08:19:13 GMT


Jason Brown commented on CASSANDRA-7203:

Hmm, interesting idea. However, I suspect any install base large enough is going to throw
a shitte tonne of cache (of the memcache/redis ilk) in front of any database, and thus probably
skewing the (read) distribution of traffic that actually makes it to the database. I have
not much solid evidence to back up this assertion, but I also feel in my gut that there is
not a one-to-one between who knocks on the front door and who actually goes to a database
(or to many of many databases).

As regards the memtable flush, you'd have to have some partitioning scheme, and flush on those
bounds - hypothetically, semi-reasonable. However, I think the gnarly work will be in the
compaction code. We now have three officially supported compaction strategies, and I wonder
how complication would be added there. Remembering what happened with incremental repair (and
the special casing of LCS vs STCS), I'd be a bit concerned the complexity creep. Perhaps this
wouldn't apply to all the strategies (I would have to think more about DateTiered), but even
that can be seen as a special casing.

At the end of the day, if you want to go down this path, sure, we can see where it leads,
and we can evaluate the results vs. the costs involved. TBH, though, this doesn't appear to
be a huge win (I think we can all agree incremental, at best), and I think we have bigger
fish to fry.

> Flush (and Compact) High Traffic Partitions Separately
> ------------------------------------------------------
>                 Key: CASSANDRA-7203
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Benedict
>              Labels: compaction, performance
> An idea possibly worth exploring is the use of streaming count-min sketches to collect
data over the up-time of a server to estimating the velocity of different partitions, so that
high-volume partitions can be flushed separately on the assumption that they will be much
smaller in number, thus reducing write amplification by permitting compaction independently
of any low-velocity data.
> Whilst the idea is reasonably straight forward, it seems that the biggest problem here
will be defining any success metric. Obviously any workload following an exponential/zipf/extreme
distribution is likely to benefit from such an approach, but whether or not that would translate
in real terms is another matter.

This message was sent by Atlassian JIRA

View raw message