lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Parsing large xml files
Date Thu, 21 May 2009 14:59:34 GMT
What fails and what is the stack trace? Have you tried just
parsing the XML in a stand-alone program independent of
indexing?

You should easily be able to parse a 50MB file with that much
memory. I suspect something else is going on here. Perhaps you're
not *really* allocating that much memory to the process. If you're
working in an IDE for instance you could be allocating memory to the
IDE but not setting the correct runtime parameters for programs
run within that IDE.

If that is irrelevant, perhaps you could add more details...

Best
Erick



On Thu, May 21, 2009 at 10:42 AM, Sudarsan, Sithu D. <
Sithu.Sudarsan@fda.hhs.gov> wrote:

>
> Hi,
>
> While trying to parse xml documents of about 50MB size, we run into
> OutOfMemoryError due to java heap space. Increasing JVM to use close 2GB
> (that is the max), does not help. Is there any API that could be used to
> handle such large single xml files?
>
> If Lucene is not the right place, please let me know alternate places to
> look for,
>
> Thanks in advance,
> Sithu D Sudarsan
> sithu.sudarsan@fda.hhs.gov
> sdsudarsan@ualr.edu
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message