lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: Too many open files when doing performance testing
Date Fri, 02 Sep 2005 04:33:29 GMT
Use that compound index format and use a single IndexSearcher instance.

Otis

--- Akhil Jain <jaina@us.ibm.com> wrote:

> 
> 
> 
> 
> 
> Actually, we might not be able to change the setting as we are using
> a
> shared server and are not allowed to change system settings.   What
> other
> things can we look at to get this fixed?
> 
> 
> Akhil Jain
> 
> 
> 
> 
>                                                                      
>  
>              Otis Gospodnetic                                        
>  
>              <otis_gospodnetic                                       
>  
>              @yahoo.com>                                             
>   To
>                                        java-user@lucene.apache.org   
>  
>              09/01/2005 09:07                                        
>   cc
>              PM                                                      
>  
>                                                                   
> Subject
>                                        Re: Too many open files when
> doing
>              Please respond to         performance testing           
>  
>                  java-user                                           
>  
>                                                                      
>  
>                                                                      
>  
>                                                                      
>  
>                                                                      
>  
>                                                                      
>  
> 
> 
> 
> 
> Hi,
> 
> 2000 doesn't sound very high.  I've used much higher values.  Because
> you have so many fields and files, you may want to increase it.  If
> you
> have Lucene in Action, we have a formula in there for calculating the
> number of open files used (section 2.7, according to
> http://www.lucenebook.com/search?query=open+files+maximum)
> 
> You said you got this error during your performance testing.  Are you
> searching a single index and using a single IndexSearcher?  Use a
> single IndexSearcher.
> 
> Otis
> 
> 
> --- "jaina (sent by Nabble.com)" <lists@nabble.com> wrote:
> 
> >
> > Thanks Otis for your questions.  Here are the answers... Please let
> > me know your thoughts....
> >
> > We do optimize() after indexWrite.
> > The maximum open file allowed is 2,000. Is this normal, too small?
> > What is your recommended number ?  We have 57 index files, how many
> > files that Lucene index will actually opens per session?
> > We use default mergeFactor 10, default merge Docs 1000.
> > We have about 18,000 documents, 23 indexed fields.
> > We are not using compound index because, but we are considering it.
> > Following is a snapshot of our typical index files:
> >
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f1
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f10
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f11
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f12
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f13
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f14
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f15
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f16
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f17
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f18
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f19
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f2
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f20
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f21
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f22
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f23
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f24
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f25
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f26
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f27
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f28
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f29
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f3
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f30
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f31
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f32
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f33
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f34
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f35
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f36
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f37
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f38
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f39
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f4
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f40
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f41
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f42
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f43
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f44
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f45
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f46
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f47
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f48
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f49
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f5
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f50
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f51
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f6
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f7
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f8
> > -rw-rw----   1 skpmq    p/gsa_cd      18719 Sep 01 09:18 _bgw.f9
> > -rw-rw----   1 skpmq    p/gsa_cd   18624312 Sep 01 09:18 _bgw.fdt
> > -rw-rw----   1 skpmq    p/gsa_cd     149752 Sep 01 09:18 _bgw.fdx
> > -rw-rw----   1 skpmq    p/gsa_cd        579 Sep 01 09:18 _bgw.fnm
> > -rw-rw----   1 skpmq    p/gsa_cd   23085952 Sep 01 09:18 _bgw.frq
> > -rw-rw----   1 skpmq    p/gsa_cd   42672641 Sep 01 09:18 _bgw.prx
> > -rw-rw----   1 skpmq    p/gsa_cd     145555 Sep 01 09:18 _bgw.tii
> > -rw-rw----   1 skpmq    p/gsa_cd   11866281 Sep 01 09:18 _bgw.tis
> > -rw-rw----   1 skpmq    p/gsa_cd          4 Sep 01 09:18 deletable
> > -rw-rw----   1 skpmq    p/gsa_cd         29 Sep 01 09:18 segments
> > --
> > Sent from the Lucene - Java Users forum at Nabble.com:
> >
>
http://www.nabble.com/Too-many-open-files-when-doing-performance-testing-t272369.html#a766958
> 
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message