lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudarsan, Sithu D." <Sithu.Sudar...@fda.hhs.gov>
Subject RE: Parsing large xml files
Date Fri, 22 May 2009 13:50:14 GMT
 
Hi Matt,

We use 32 bit JVM. Though it is supposed to have upto 4GB, any
assignment above 2GB in Windows XP fails. The machine has  quad-core
dual processor.

On Linux we're able to use 4GB though!

If there is any setting that will let us use 4GB do let me know.

Thanks,
Sithu D Sudarsan

-----Original Message-----
From: Matthew Hall [mailto:mhall@informatics.jax.org] 
Sent: Friday, May 22, 2009 8:59 AM
To: java-user@lucene.apache.org
Subject: Re: Parsing large xml files

2g... should not be a maximum for any Jvm that I know of.

Assuming you are running a 32 bit Jvm you are actually able to address a

bit under 4G of memory, I've always used around 3.6G when trying to max 
out a 32 bit jvm.  Technically speaking it should be able to address 4g 
under a 32 bit or, however a certain percentage of the memory is set 
aside for overhead, so you can only really use a bit less than the max.

If you have a 64 bit os/jvm (which you likely might), you can use the 
-d64 setting for your runtime environment to set your maximum memory 
much.. MUCH higher, for example we regularly use 6G of memory on our 
application servers here at the lab.

Hope this helps you a bit,

Matt

crackeur@comcast.net wrote:
> http://vtd-xml.sf.net 
>
>
> ----- Original Message ----- 
> From: "Sithu D. Sudarsan" <Sithu.Sudarsan@fda.hhs.gov> 
> To: java-user@lucene.apache.org 
> Sent: Thursday, May 21, 2009 7:42:59 AM GMT -08:00 US/Canada Pacific 
> Subject: Parsing large xml files 
>
>
> Hi, 
>
> While trying to parse xml documents of about 50MB size, we run into 
> OutOfMemoryError due to java heap space. Increasing JVM to use close
2GB 
> (that is the max), does not help. Is there any API that could be used
to 
> handle such large single xml files? 
>
> If Lucene is not the right place, please let me know alternate places
to 
> look for, 
>
> Thanks in advance, 
> Sithu D Sudarsan 
> sithu.sudarsan@fda.hhs.gov 
> sdsudarsan@ualr.edu 
>
>
>
>
>   



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message