lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trieu, Jason T" <Trieu.Ja...@con-way.com>
Subject How to determine memory required for searching
Date Thu, 04 Aug 2011 21:41:56 GMT
Hello,

I keep getting java.lang.OutOfMemoryError while doing search and would like to get some ideas
to resolve the issue..

BACKGROUND:

IBM AIX server with 8 GIG of memory.
Lucene index contains 625 million documents with each having roughly 200 bytes of stored fields/data.
Running Lucene IndexSearcher  in JVM with -Xms3072m -Xmx3072m (more  memory than that then
the system kills the JVM).

PROBLEM:

I can search successfully search the index if I do not user sort or filtering.   Although
I need to invoke IndexReader with a termInfosIndexDivisor of at least 100 (IndexReader.open(dir,NoDeletionPolicy.INSTANCE,true,100)
If I include a sort ( of a numeric field) or a numeric range filter, I get OutOfMemoryError
no matter how big the termInfosIndexDivisor is.  I've tried it as big as 100,000,000 and still
got OutOfMemoryError.
It seems I am running against hard limits of resources.  Perhaps 8 GB of memory is just simply
not enough to handle an index of 600 million documents.  But before telling management that
they must get more memory, I'd to see if there might be other ways to accomplish this.
Thanks in advance.

Jason





Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message