lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konrad Kolosowski" <konr...@ca.ibm.com>
Subject Re: OutOfMemoryErrors searching with WildCardQueries
Date Thu, 12 Jun 2003 22:53:03 GMT
After Dave Kor put me on track, I thought I will need to dive into hacking
Lucene on my own, but having the fix already in the repository is great.
Thank you Doug.
I assume the fix will be picked up by 1.3 release.  Is there an expected
time frame for 1.3 Final build?
Thanks.

Konrad Kolosowski



                                                                                         
                                             
                      Doug Cutting                                                       
                                             
                      <cutting@lucene.c        To:       Lucene Users List <lucene-user@jakarta.apache.org>
                           
                      om>                      cc:                                    
                                                
                                               Subject:  Re: OutOfMemoryErrors searching with
WildCardQueries                          
                      06/12/2003 02:28                                                   
                                             
                      PM                                                                 
                                             
                      Please respond to                                                  
                                             
                      "Lucene Users                                                      
                                             
                      List"                                                              
                                             
                                                                                         
                                             



Konrad Kolosowski wrote:
> If the index grows to hundred thousand documents, with users
simultaneously
> searching indexes for different locales, what is the best way to cup the
> memory requirement?  Limiting number of terms, or number of terms
> containing wild cards, or eliminating wild card searches altogether.

This was discussed recently on lucene-dev@jakarta.apache.org in a thread
whose subject contains "too many hits - OutOfMemoryError".

I checked in a patch which limits the number of terms that a wildcard is
permitted to expand into.  The default is 1000.  If a term expands to
more than that then an exception is thrown.  Each term that a wildcard
expands into requires around 2kB.  So this limits each wildcarded query
term to 2MB.  If you have queries with large numbers of wildcarded terms
then you might consider also limiting that.

This patch is in the latest version of Lucene in CVS, but not yet in a
release.

Doug


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message