lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David_Birthw...@VWR.COM
Subject Re: too many hits - OutOfMemoryError
Date Wed, 28 May 2003 15:50:07 GMT


When performing wildcard queries, the bulk of the memory is used during
wildcard term expansion.  The memory requirement is proportional to the
number of matching terms, not the number of hits.

You should make sure you are using the latest Lucene.  There was a fix in
1.3 to reduce the memory requirements of all all queries.

But, wildcard queries that expand to many terms are allways going to be
memory intensive in Lucene.  We ran into this problem and decided to put a
check on the number of expanded terms and abort the query if the number got
too high.  If you're ambitious, you could modify the Lucene source to
serialize the query process for queries with a large number of terms, but
that would be a bit of work.  If you absolutely require these huge wildcard
queries, then you may have to look into it, though.

Non-wildcard queries that return a large result set should not be a memory
problem, though.


                      Cory Albright                                                      
                      <albright@mt.sri.        To:       "Lucene Users List"          
                      com>                      <>
                      05/28/03 11:16 AM        Subject:  Re: too many hits - OutOfMemoryError
                      Please respond to                                                  
                      "Lucene Users                                                      

Yes. Is that the problem?

At 05:13 PM 5/28/2003 +0200, you wrote:
> > When I search with a query I know will hit most of the 1.8 million
> > records, the "collect" print
> > does not even print, it eats up the 700+MB I allocated and then
> > throws an OutOfMemoryError.
>Are you using wildcard queries?
>Eric Jain
>To unsubscribe, e-mail:
>For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message