Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of prasanna.pradhan@gmail.com
 designates 74.125.92.27 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=itHhyORq5DQSKDe6KT5lrx4hyaKo+/Qvf2NmxzjslRBKBx046NQNiuZ3Spt9QVZ4tX
         TzB1gHS4X3r0d4MMkJv/edSQfx2kyrxbXifQOwL4oWumaACjxuMoov+poqmZd7GdrdZ6
         vL0nmo02QwWvw+jPvmWE0FEIzcDFpx7xy2zKw=
MIME-Version: 1.0
In-Reply-To: <F8A04958BB57B64096F9BE6B9572564304C73C12@FMD3VS012.fda.gov>
References: <F8A04958BB57B64096F9BE6B9572564304C73C12@FMD3VS012.fda.gov>
Date: Fri, 22 May 2009 23:34:45 +0530
Message-ID: <2e3c5ba70905221104u50b747e6rc56764d1a15839f4@mail.gmail.com>
Subject: Re: Parsing large xml files
From: prasanna pradhan <prasanna.pradhan@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=00163646c2e27bca79046a841817

--00163646c2e27bca79046a841817
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

We had similar a problem  where we had to parse 1 GB XML files.Better
transform to array like json and write a custom search API using lucene.

On Thu, May 21, 2009 at 8:12 PM, Sudarsan, Sithu D. <
Sithu.Sudarsan@fda.hhs.gov> wrote:

>
> Hi,
>
> While trying to parse xml documents of about 50MB size, we run into
> OutOfMemoryError due to java heap space. Increasing JVM to use close 2GB
> (that is the max), does not help. Is there any API that could be used to
> handle such large single xml files?
>
> If Lucene is not the right place, please let me know alternate places to
> look for,
>
> Thanks in advance,
> Sithu D Sudarsan
> sithu.sudarsan@fda.hhs.gov
> sdsudarsan@ualr.edu
>
>
>
>


-- 
Thanks,
Prasanna

--00163646c2e27bca79046a841817--