cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12245) initial view build can be parallel
Date Wed, 20 Jul 2016 10:43:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385657#comment-15385657
] 

Sylvain Lebresne commented on CASSANDRA-12245:
----------------------------------------------

bq. just iterate through sstables, not worrying about duplicates, and include the timestamp
of the original write in the MV mutation.

I could misunderstand this but if you talking about dealing with each sstable completely individually,
I don't think we can do that. It's not that we have to merge sstables to avoid dealing with
duplicates, it's we need to current full version of a row to create the proper MV entry. For
instance, say you have a first insert to a row in one sstable that contains the following
columns/values {{c1=2, c2=3, c3='foo'}}, and a later update to that row in another sstable
containing {{c1=4}}. And further say that {{c1}} is part of your view PK. If we deal with
both sstable individually, in the first case we'll insert an entry for {{c1=2}}, with the
other value, while for the 2nd sstable we'll create a *different* entry for {{c1=4}} with
no other values. Which is doubly wrong as the {{c1=2}} entry shouldn't exist, and the {{c1=4}}
entry should have the other values for {{c2}} and {{c3}}.

With that said, several weeks to build a view over 3TB is indeed not ideal, and we can indeed
at least do vnodes in parallel. 

> initial view build can be parallel
> ----------------------------------
>
>                 Key: CASSANDRA-12245
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12245
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Tom van der Woerdt
>
> On a node with lots of data (~3TB) building a materialized view takes several weeks,
which is not ideal. It's doing this in a single thread.
> There are several potential ways this can be optimized :
>  * do vnodes in parallel, instead of going through the entire range in one thread
>  * just iterate through sstables, not worrying about duplicates, and include the timestamp
of the original write in the MV mutation. since this doesn't exclude duplicates it does increase
the amount of work and could temporarily surface ghost rows (yikes) but I guess that's why
they call it eventual consistency. doing it this way can avoid holding references to all tables
on disk, allows parallelization, and removes the need to check other sstables for existing
data. this is essentially the 'do a full repair' path



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message