lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David_Birthw...@VWR.COM
Subject Re: too many hits - OutOfMemoryError
Date Wed, 28 May 2003 15:50:07 GMT

Cory,

When performing wildcard queries, the bulk of the memory is used during
wildcard term expansion.  The memory requirement is proportional to the
number of matching terms, not the number of hits.

You should make sure you are using the latest Lucene.  There was a fix in
1.3 to reduce the memory requirements of all all queries.

But, wildcard queries that expand to many terms are allways going to be
memory intensive in Lucene.  We ran into this problem and decided to put a
check on the number of expanded terms and abort the query if the number got
too high.  If you're ambitious, you could modify the Lucene source to
serialize the query process for queries with a large number of terms, but
that would be a bit of work.  If you absolutely require these huge wildcard
queries, then you may have to look into it, though.

Non-wildcard queries that return a large result set should not be a memory
problem, though.

Dave




                                                                                         
          
                      Cory Albright                                                      
          
                      <albright@mt.sri.        To:       "Lucene Users List"          
             
                      com>                      <lucene-user@jakarta.apache.org>
                   
                                               cc:                                       
          
                      05/28/03 11:16 AM        Subject:  Re: too many hits - OutOfMemoryError
      
                      Please respond to                                                  
          
                      "Lucene Users                                                      
          
                      List"                                                              
          
                                                                                         
          
                                                                                         
          




Yes. Is that the problem?

At 05:13 PM 5/28/2003 +0200, you wrote:
> > When I search with a query I know will hit most of the 1.8 million
> > records, the "collect" print
> > does not even print, it eats up the 700+MB I allocated and then
> > throws an OutOfMemoryError.
>
>Are you using wildcard queries?
>
>--
>Eric Jain
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org







---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message