Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Message-ID: <4201924.158041295885744495.JavaMail.jira@thor>
Date: Mon, 24 Jan 2011 11:15:44 -0500 (EST)
From: "Nick Bailey (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Subject: [jira] Commented: (CASSANDRA-1898) json2sstable should support
 streaming
In-Reply-To: <12589909.1031293128208162.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985778#action_12985778 ] 

Nick Bailey commented on CASSANDRA-1898:
----------------------------------------

It certainly shouldn't error. I think the fix could be with with either export or import though.  

Is having a comma in an object with no attribute like that technically valid json?  If so I would say json2sstable should handle that case and skip it. If not it might be easier to just fix sstable2json to not output that comma.

Or maybe they should both be fixed.

> json2sstable should support streaming
> -------------------------------------
>
>                 Key: CASSANDRA-1898
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1898
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Nick Bailey
>            Assignee: Pavel Yaskevich
>             Fix For: 0.7.1
>
>         Attachments: CASSANDRA-1898-v2.patch, CASSANDRA-1898-v3.patch, CASSANDRA-1898-v4.patch, CASSANDRA-1898.patch
>
>   Original Estimate: 8h
>          Time Spent: 8h
>  Remaining Estimate: 0h
>
> json2sstable loads the entire json file into memory. This is so it can sort the file before creating an sstable. If the file was created using sstable2json and the partitioner isn't changing, this isn't necessary.  For very large files this means json2sstable requires a huge amount of memory.
> There should be an option to stream the file. A simple check for out of order keys will prevent writing bad sstables.
> This should be possible with the SAX style parser available in our current json library.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.