cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11875) Create sstableconvert tool with support to ma format
Date Sun, 29 May 2016 17:35:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306004#comment-15306004
] 

Paulo Motta commented on CASSANDRA-11875:
-----------------------------------------

Thanks for the update! See follow-up comments below:

* There is still some common code between {{StandaloneConverter}} and {{StandaloneUpgrader}}
main methods, most of it can be extracted to common methods use by both.
* The supported version check will probably be used in other places, so we should probably
move it to {{BigVersion}}. There are already {{isCompatible}} and  {{isCompatibleForStream}}
methods, so we can maybe add a {{isCompatibeForWriting}}. It would be nice if you could add
a unit test that checks that trying to converting to an unsupported version should fail.

bq. Also added a new Test suit SSTableConversionTest, but it has some problem when performing
conversion.

It seems this is due to re-loading the schema on {{Schema.instance.loadFromDisk(false)}} on
{{StandaloneConverter}}. Since we want to focus testing the conversion itself, we will probably
have more flexibility testing the internal class {{SSTableConverter}}, so we don't have to
add special options for testing on {{StandaloneConverter}} and can also play around with SSTableReaders
directly. 

Instead of basing our tests on {{SSTableRewriterTest}} as initially discussed, it's probably
more convenient to base it on {{CQLTester}} since we will be doing data conversions and testing
this at a higher level with CQL is the only way to ensure converted data is being interpreted
correctly. The tests should have more or less the following structure:

* Insert Data with CQL
* Flush to disk
* Read and verify data in current version with CQL
* Keep reference to original SSTableReaders and cleanup ColumnFamilyStore (clearUnsafe)
* Perform conversion on original SSTableReaders
* Verify metadata was converted correctly on converted SSTableReaders 
* Add converted SSTableReaders to ColumnFamilyStore (addSSTable)
* Read and verify data in converted version with CQL

For the mb to ma conversion, since there is no data conversion involved (only metadata), you
can use {{SimpleQueryTest.testTableWithoutClustering}} as a example to write the first test
case. You may use utility {{CQLTester}} utility methods such as {{flush()}} and {{getCurrentColumnFamilyStore()}}
to flush and access {{ColumnFamilyStore}}.

> Create sstableconvert tool with support to ma format
> ----------------------------------------------------
>
>                 Key: CASSANDRA-11875
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
>             Project: Cassandra
>          Issue Type: Sub-task
>          Components: Tools
>            Reporter: Paulo Motta
>            Assignee: Kaide Mu
>            Priority: Minor
>         Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes into {{BigFormat.getLatestVersion()}}.
We should generalize it by making it receive a {{target}} version and probably also rename
it to {{SSTableConverter}}. 
> Based on this we can create an {{StandaloneDowngrader}} tool which will perform downgrade
of specified sstables to a target version. To start with, we should support only downgrading
to {{ma}} format (from current format {{mb}}), downgrade to any other version should be forbidden.
Since we already support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the current format,
perform the downgrade, and verify data in the new format is correct. This will be the base
tests suite for more advanced conversions in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message