Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Tue, 2 Dec 2014 08:19:13 +0000 (UTC)
From: "Jason Brown (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12713416.1399677113000.56280.1417508353147@Atlassian.JIRA>
In-Reply-To: <JIRA.12713416.1399677113000@Atlassian.JIRA>
References: <JIRA.12713416.1399677113000@Atlassian.JIRA>
 <JIRA.12713416.1399677113243@arcas>
Subject: [jira] [Commented] (CASSANDRA-7203) Flush (and Compact) High
 Traffic Partitions Separately
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231151#comment-14231151 ] 

Jason Brown commented on CASSANDRA-7203:
----------------------------------------

Hmm, interesting idea. However, I suspect any install base large enough is going to throw a shitte tonne of cache (of the memcache/redis ilk) in front of any database, and thus probably skewing the (read) distribution of traffic that actually makes it to the database. I have not much solid evidence to back up this assertion, but I also feel in my gut that there is not a one-to-one between who knocks on the front door and who actually goes to a database (or to many of many databases).

As regards the memtable flush, you'd have to have some partitioning scheme, and flush on those bounds - hypothetically, semi-reasonable. However, I think the gnarly work will be in the compaction code. We now have three officially supported compaction strategies, and I wonder how complication would be added there. Remembering what happened with incremental repair (and the special casing of LCS vs STCS), I'd be a bit concerned the complexity creep. Perhaps this wouldn't apply to all the strategies (I would have to think more about DateTiered), but even that can be seen as a special casing.

At the end of the day, if you want to go down this path, sure, we can see where it leads, and we can evaluate the results vs. the costs involved. TBH, though, this doesn't appear to be a huge win (I think we can all agree incremental, at best), and I think we have bigger fish to fry.

> Flush (and Compact) High Traffic Partitions Separately
> ------------------------------------------------------
>
>                 Key: CASSANDRA-7203
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7203
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Benedict
>              Labels: compaction, performance
>
> An idea possibly worth exploring is the use of streaming count-min sketches to collect data over the up-time of a server to estimating the velocity of different partitions, so that high-volume partitions can be flushed separately on the assumption that they will be much smaller in number, thus reducing write amplification by permitting compaction independently of any low-velocity data.
> Whilst the idea is reasonably straight forward, it seems that the biggest problem here will be defining any success metric. Obviously any workload following an exponential/zipf/extreme distribution is likely to benefit from such an approach, but whether or not that would translate in real terms is another matter.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)