cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Stupp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11206) Support large partitions on the 3.0 sstable format
Date Thu, 14 Apr 2016 21:48:25 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241975#comment-15241975
] 

Robert Stupp commented on CASSANDRA-11206:
------------------------------------------

bq. need to change the version of sstable
The change does not change the index sstable format - just the format of the saved key cache.

bq. AutoSavingCache change require a step on the users part
No, all that happens is that you lose the contents of the old saved key cache. This is since
the change requires some more information on shallow indexed entries (offset in index file).

bq. 0,1,2 magic bytes
Made these constants and pushed a commit for this.

bq. dtests/unit test with column_index_cache_size_in_kb: 0
I've setup a new branch {{11206-large-part-0kb-trunk}} and triggered CI for this. [testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-11206-large-part-0kb-trunk-testall/lastBuild/]
[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-11206-large-part-0kb-trunk-dtest/lastBuild/]


> Support large partitions on the 3.0 sstable format
> --------------------------------------------------
>
>                 Key: CASSANDRA-11206
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11206
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Robert Stupp
>             Fix For: 3.x
>
>         Attachments: 11206-gc.png, trunk-gc.png
>
>
> Cassandra saves a sample of IndexInfo objects that store the offset within each partition
of every 64KB (by default) range of rows.  To find a row, we binary search this sample, then
scan the partition of the appropriate range.
> The problem is that this scales poorly as partitions grow: on a cache miss, we deserialize
the entire set of IndexInfo, which both creates a lot of GC overhead (as noted in CASSANDRA-9754)
but is also non-negligible i/o activity (relative to reading a single 64KB row range) as partitions
get truly large.
> We introduced an "offset map" in CASSANDRA-10314 that allows us to perform the IndexInfo
bsearch while only deserializing IndexInfo that we need to compare against, i.e. log(N) deserializations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message