Return-Path: Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: (qmail 80460 invoked from network); 12 Jan 2011 18:59:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 12 Jan 2011 18:59:12 -0000 Received: (qmail 86001 invoked by uid 500); 12 Jan 2011 18:59:11 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 85972 invoked by uid 500); 12 Jan 2011 18:59:11 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 85964 invoked by uid 99); 12 Jan 2011 18:59:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 18:59:11 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 18:59:09 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id p0CIwlBR004714 for ; Wed, 12 Jan 2011 18:58:47 GMT Message-ID: <16361682.319661294858727272.JavaMail.jira@thor> Date: Wed, 12 Jan 2011 13:58:47 -0500 (EST) From: "Pavel Yaskevich (JIRA)" To: commits@cassandra.apache.org Subject: [jira] Commented: (CASSANDRA-1898) json2sstable should support streaming In-Reply-To: <12589909.1031293128208162.JavaMail.jira@thor> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/CASSANDRA-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980859#action_12980859 ] Pavel Yaskevich commented on CASSANDRA-1898: -------------------------------------------- Thanks Brandon! I see now that this is happens because Jackson wants to read a row after ',' but there is nothing left to read. SSTableExport generates following {code} { "726f7741": [["6e616d65", "76616c", 1, false]] , } {code} I will fix this ASAP. > json2sstable should support streaming > ------------------------------------- > > Key: CASSANDRA-1898 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1898 > Project: Cassandra > Issue Type: Improvement > Components: Tools > Reporter: Nick Bailey > Assignee: Pavel Yaskevich > Fix For: 0.7.1 > > Attachments: CASSANDRA-1898.patch > > Original Estimate: 8h > Time Spent: 8h > Remaining Estimate: 0h > > json2sstable loads the entire json file into memory. This is so it can sort the file before creating an sstable. If the file was created using sstable2json and the partitioner isn't changing, this isn't necessary. For very large files this means json2sstable requires a huge amount of memory. > There should be an option to stream the file. A simple check for out of order keys will prevent writing bad sstables. > This should be possible with the SAX style parser available in our current json library. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.