lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: How to determine memory required for searching
Date Fri, 05 Aug 2011 09:15:37 GMT
> Running Lucene IndexSearcher  in JVM with -Xms3072m -Xmx3072m (more  memory than that
then the system kills the JVM).

I'd try and fix that first.  Use a different JVM?  No point in buying
more memory if you can't even use all you've got now.

What version of lucene are you using?
http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html
implies you might be better off with a bleeding edge version.

Or maybe you need to look at splitting your index.

--
Ian.


On Thu, Aug 4, 2011 at 10:41 PM, Trieu, Jason T <Trieu.Jason@con-way.com> wrote:
> Hello,
>
> I keep getting java.lang.OutOfMemoryError while doing search and would like to get some
ideas to resolve the issue..
>
> BACKGROUND:
>
> IBM AIX server with 8 GIG of memory.
> Lucene index contains 625 million documents with each having roughly 200 bytes of stored
fields/data.
> Running Lucene IndexSearcher  in JVM with -Xms3072m -Xmx3072m (more  memory than that
then the system kills the JVM).
>
> PROBLEM:
>
> I can search successfully search the index if I do not user sort or filtering.   Although
I need to invoke IndexReader with a termInfosIndexDivisor of at least 100 (IndexReader.open(dir,NoDeletionPolicy.INSTANCE,true,100)
> If I include a sort ( of a numeric field) or a numeric range filter, I get OutOfMemoryError
no matter how big the termInfosIndexDivisor is.  I've tried it as big as 100,000,000 and
still got OutOfMemoryError.
> It seems I am running against hard limits of resources.  Perhaps 8 GB of memory is just
simply not enough to handle an index of 600 million documents.  But before telling management
that they must get more memory, I'd to see if there might be other ways to accomplish this.
> Thanks in advance.
>
> Jason
>
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message