cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-2677) Optimize streaming to be single-pass
Date Wed, 13 Jul 2011 16:27:00 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis resolved CASSANDRA-2677.
---------------------------------------

    Resolution: Fixed
      Reviewer: jbellis
      Assignee: Yuki Morishita

Committed.  Nice work, Yuki!

> Optimize streaming to be single-pass
> ------------------------------------
>
>                 Key: CASSANDRA-2677
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2677
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.0
>
>         Attachments: trunk-2677.txt
>
>
> Streaming currently is a two-pass operation: one to write the Data component do disk
from the socket, then another to build the index and bloom filter from it.  This means we
do about 2x the i/o we would if we created the index and BF during the original write.
> For node movement this was not considered to be a Big Deal because the stream target
is not a member of the ring, so we can be inefficient without hurting live queries.  But optimizing
node movement to not require un/rebootstrap (CASSANDRA-1427) and bulk load (CASSANDRA-1278)
mean we can stream to live nodes too.
> The main obstacle here is we don't know how many keys will be in the new sstable ahead
of time, which we need to size the bloom filter correctly. We can solve this by including
that information (or a close approximation) in the stream setup -- the source node can calculate
that without hitting disk from the in-memory index summary.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message