cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-10990) Support streaming of older version sstables in 3.0
Date Sat, 05 Mar 2016 02:52:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181464#comment-15181464
] 

Paulo Motta edited comment on CASSANDRA-10990 at 3/5/16 2:52 AM:
-----------------------------------------------------------------

Updated branch with reworked {{RewindableDataInputStreamPlus}}, now treating the spill file
as circular buffer with max size. Now required space for spill buffer file of legacy {{StreamDeserializer}}
is {{max(sstableSize, MAX_INT)}}, capping the max required space (and max partition size)
for streaming legacy/compact sstables on ~2GB. I also updated documentation and tests. WDYT
[~yukim]?

Besides that, I added a new [dtest|https://github.com/pauloricardomg/cassandra-dtest/commit/b59ea329a5b9302372c57956f7015111fd47a4ae]
for testing repair of older version sstables on 3.0+, and it seems to work. Now we have the
following dtests:
* [bootstrap dtests|https://github.com/pauloricardomg/cassandra-dtest/blob/10990/upgrade_8099_test.py#L340]
* [sstable loader dtests|https://github.com/pauloricardomg/cassandra-dtest/blob/10990/sstable_generation_loading_test.py#L181]
* [repair dtests|https://github.com/pauloricardomg/cassandra-dtest/commit/b59ea329a5b9302372c57956f7015111fd47a4ae]

For simplicity, I opted for writing the old streamed sstables in the new format without {{EncodingStats}}
(since there is no {{SerializationHeader}} available), what may make these sstables less optimized
in terms of storage space from what I understood. Do you think we should construct these stats
when receiving the sstable or not bother [~slebresne] ? Also, do you recall of any other edge
case we should watch here?

Thanks!


was (Author: pauloricardomg):
Updated branch with reworked {{RewindableDataInputStreamPlus}}, now treating the spill file
as circular buffer with max size. Now required space for spill buffer file of legacy {{StreamDeserializer}}
is {{max(sstableSize, MAX_INT)}}, capping the max required space for streaming older version
sstables on ~2GB. I also updated documentation and tests. WDYT [~yukim]?

Besides that, I added a new [dtest|https://github.com/pauloricardomg/cassandra-dtest/commit/b59ea329a5b9302372c57956f7015111fd47a4ae]
for testing repair of older version sstables on 3.0+, and it seems to work. Now we have the
following dtests:
* [bootstrap dtests|https://github.com/pauloricardomg/cassandra-dtest/blob/10990/upgrade_8099_test.py#L340]
* [sstable loader dtests|https://github.com/pauloricardomg/cassandra-dtest/blob/10990/sstable_generation_loading_test.py#L181]
* [repair dtests|https://github.com/pauloricardomg/cassandra-dtest/commit/b59ea329a5b9302372c57956f7015111fd47a4ae]

For simplicity, I opted for writing the old streamed sstables in the new format without {{EncodingStats}}
(since there is no {{SerializationHeader}} available), what may make these sstables less optimized
in terms of storage space from what I understood. Do you think we should construct these stats
when receiving the sstable or not bother [~slebresne] ? Also, do you recall of any other edge
case we should watch here?

Thanks!

> Support streaming of older version sstables in 3.0
> --------------------------------------------------
>
>                 Key: CASSANDRA-10990
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10990
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>            Reporter: Jeremy Hanna
>            Assignee: Paulo Motta
>
> In 2.0 we introduced support for streaming older versioned sstables (CASSANDRA-5772).
 In 3.0, because of the rewrite of the storage layer, this became no longer supported.  So
currently, while 3.0 can read sstables in the 2.1/2.2 format, it cannot stream the older versioned
sstables.  We should do some work to make this still possible to be consistent with what CASSANDRA-5772
provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message