cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Evans <eev...@rackspace.com>
Subject Re: Heap sudden jump during import
Date Wed, 07 Apr 2010 20:51:28 GMT
On Tue, 2010-04-06 at 10:55 -0700, Tatu Saloranta wrote:
> On Tue, Apr 6, 2010 at 12:15 AM, JKnight JKnight <beuknight@gmail.com>
> wrote:
> > When import, all data in json file will load in memory. So that, you
> can not
> > import large data.
> > You need to export large sstable file to many small json files, and
> run
> > import.
> 
> Why would you ever read the whole file in memory? JSON is very easily
> streamable. Or does the whole data set need to be validated or
> something (I assume not, if file splitting could be used). Perhaps it
> is just an implementation flaw in importer tool. 

It's been awhile, but if I'm not mistaken, this is because we're writing
SSTables and the records must be written in decorated-key sorted order.

-- 
Eric Evans
eevans@rackspace.com


Mime
View raw message