Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of alienth@gmail.com designates
 209.85.210.60 as permitted sender)
MIME-Version: 1.0
Date: Sun, 13 Mar 2011 03:53:02 -0700 (PDT)
In-Reply-To: <AANLkTin_wZD8FUyXRzPa4uFQ+8OAPFBv7F+xhO0O-GKd@mail.gmail.com>
References: <AANLkTin_wZD8FUyXRzPa4uFQ+8OAPFBv7F+xhO0O-GKd@mail.gmail.com>
User-Agent: G2/1.0
Message-ID: 
 <63c4089f-b8c8-43c1-8db0-d1ae77d29506@d12g2000prj.googlegroups.com>
Subject: Re: json2sstable hanging on large sstable2json-generated JSON file
From: Jason Harvey <alienth@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

It eventually died with an OOM error. Guess the table was just too
big :( Created an improvement request ticket:
https://issues.apache.org/jira/browse/CASSANDRA-2322

Jason

On Mar 12, 10:50=A0pm, Jason Harvey <alie...@gmail.com> wrote:
> Trying to import a 3GB JSON file which was exported from sstable2json.
> I let it run for over an hour and saw zero IO activity. The last thing
> it logs is the following:
>
> DEBUG 23:19:32,638 collecting 0 of 2147483647:
> Avro/Schema:false:2042@1298067089267
> DEBUG 23:19:32,638 collecting 1 of 2147483647: reddit:false:2502@12980670=
89267
>
> Considering I saw zero reads on my disk when I ran it, I don't think
> it is even reading the JSON file.
>
> I shrunk the file down to a handful of keys, and it worked fine. Is
> there an issue with json2sstable loading large JSON files? Does it try
> to read it into memory?
>
> Also as a note, this data is unsorted. I did generate it via
> sstable2json, but my sstables were broken and had unsorted data, which
> is the whole reason I am doing this.
>
> Thanks!
> Jason