cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Bailey (JIRA)" <j...@apache.org>
Subject [jira] Created: (CASSANDRA-1898) json2sstable should support streaming
Date Thu, 23 Dec 2010 18:16:48 GMT
json2sstable should support streaming
-------------------------------------

                 Key: CASSANDRA-1898
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1898
             Project: Cassandra
          Issue Type: Improvement
          Components: Tools
    Affects Versions: 0.7.0 rc 2, 0.7.0
            Reporter: Nick Bailey
             Fix For: 0.7.1


json2sstable loads the entire json file into memory. This is so it can sort the file before
creating an sstable. If the file was created using sstable2json and the partitioner isn't
changing, this isn't necessary.  For very large files this means json2sstable requires a huge
amount of memory.

There should be an option to stream the file. A simple check for out of order keys will prevent
writing bad sstables.

This should be possible with the SAX style parser available in our current json library.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message