lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harini Raghavan" <harini.ragha...@insideview.com>
Subject Searching with too many clauses + Out of Memory
Date Wed, 01 Aug 2007 08:44:16 GMT
 Hi Everyone,

I am using Compass 1.1 M2 which supports Lucene 2.2 to store & search huge
amount of company, executive and employment data. There are some usecases
where I need to search for executives/employments on the result set of
company search. But when I try to create a compass query to search for
executives for over 1 lac company ids, it runs out of memory as the query is
huge. Here is the exception stack trace:

java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.lucene.index.SegmentReader.termDocs(SegmentReader.java:342)
at org.apache.lucene.index.MultiTermDocs.termDocs(MultiReader.java:435)
at org.apache.lucene.index.MultiTermDocs.termDocs(MultiReader.java:428)
at org.apache.lucene.index.MultiTermDocs.read(MultiReader.java:393)
at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
at org.apache.lucene.util.ScorerDocQueue.topNextAndAdjustElsePop(
ScorerDocQueue.ja va:116)
at org.apache.lucene.search.DisjunctionSumScorer.advanceAfterCurrent(DisjunctionSu
mScorer.java:175)
at org.apache.lucene.search.DisjunctionSumScorer.next(
DisjunctionSumScorer.java:14 6)
at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:327)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:124)
at org.apache.lucene.search.MultiSearcher.search(MultiSearcher.java:232)
at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:74)
at org.apache.lucene.search.Hits.<init>(Hits.java:61)
at org.apache.lucene.search.Searcher.search(Searcher.java:55)
at
org.compass.core.lucene.engine.transaction.ReadCommittedTransaction.findByQuery(
ReadCommittedTransaction.java:469)
at
org.compass.core.lucene.engine.transaction.ReadCommittedTransaction.doFind(Read
CommittedTransaction.java:426)
at org.compass.core.lucene.engine.transaction.AbstractTransaction.find(AbstractTra
nsaction.java:91)
at org.compass.core.lucene.engine.LuceneSearchEngine.find(
LuceneSearchEngine.java: 379)
at org.compass.core.lucene.engine.LuceneSearchEngineQuery.hits(LuceneSearchEngineQ
uery.java:151)
at org.compass.core.impl.DefaultCompassQuery.hits(DefaultCompassQuery.java:133)

at org.compass.core.support.search.CompassSearchHelper.performSearch(CompassSearch
Helper.java:144)
at org.compass.core.support.search.CompassSearchHelper$1.doInCompass(CompassSearch
Helper.java:89)
at org.compass.core.CompassTemplate.execute(CompassTemplate.java:137)
at org.compass.core.support.search.CompassSearchHelper.search(CompassSearchHelper.
java:86)

It looks like this error is actually in the lucene code. It would be great
if anyone in this group has an idea about this kind of usecase and has some
suggestions.

Thanks,
Harini

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message