lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Benge" <todd.be...@gmail.com>
Subject OutOfMemory Problems Lucene 2.4 / Tomcat
Date Wed, 29 Oct 2008 22:03:45 GMT
Hi,

I'm the lead engineer for search on a large website using lucene for search.

We're indexing about 300M documents in ~ 100 indices.  The indices add
 up to ~ 60G.

The indices are sorted into 4 different Multisearcher with the largest
handling ~50G.

The code is basically like the following:

private static MultiSearcher searcher;

public void init(File files) {

     IndexSearcer [] searchers = new IndexSearcher[files.length] ();
     int i = 0;
     for ( File file: files ) {
          searchers[i++] = new IndexSearcher(FSDirectory.getDirectory(file);
     }

searcher = new MultiSearcher(searchers);
}

public Searcher getSearcher() {
   return searcher;
}

We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
Performance is good but servers are consistently hanging with
OutOfMemory errors.

We're allocating 4G in the heap to each server.

Is there any way to control the amount of memory Lucene consume for
caching?  Any other suggestions on fixing the memory errors?

Thanks,

Todd

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message