lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Smith <ssm...@mainstreamdata.com>
Subject RE: Lucene slow performance
Date Fri, 15 Mar 2013 23:54:23 GMT
To answer your first question: "good guess" :-). Yes, this is running on windows.  Sorry, I
should have mentioned this.

Your second point was very interesting.  My assumption was that the IndexReader would get
closed when the garbage collector realized that these objects were no longer being used. 
I use openIfChanged() to get the new IndexReader.  But, I don't do a close() on the previous
reader.

-----Original Message-----
From: Uwe Schindler [mailto:uwe@thetaphi.de] 
Sent: Friday, March 15, 2013 5:29 PM
To: java-user@lucene.apache.org; simon.willnauer@gmail.com
Subject: RE: Lucene slow performance

OK, your configuration seems fine. I would have the following idea:
- Are you using windows? If yes, then IndexWriter cannot remove unused files when they are
still in use (e.g. hold by an open IndexReader)
- When you get a new IndexReader after changes to the index, do you close the old ones? If
not, the above will prevent IndexWriter from removing older cfs files. They are no longer
used, but linger around in filesystem. Because The older IndexReader stay open forever (if
you missed to close them), IndexWriter troies serveral times to delete them, but never succeeds.
On Unix/Linux, open files can be deleted, not on windows.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Simon Willnauer [mailto:simon.willnauer@gmail.com]
> Sent: Saturday, March 16, 2013 12:08 AM
> To: java-user@lucene.apache.org
> Subject: Re: Lucene slow performance
> 
> On Sat, Mar 16, 2013 at 12:02 AM, Scott Smith 
> <ssmith@mainstreamdata.com> wrote:
> > " Do you always close IndexWriter after adding few documents and 
> > when
> closing, disable "wait for merge"? In that case, all merges are 
> interrupted and the merge policy never has a chance to merge at all 
> (because you are opening and closing IndexWriter all the time with cancelling all merges)?"
> >
> > Frankly I don't quite understand what this means.  When I "close" 
> > the
> indexwriter, I simply call close().  Is that the wrong thing?
> that should be fine...
> 
> this sounds very odd though, do you see file that get actually removed 
> / merged if you call IndexWriter#forceMerge(1)
> 
> simon
> >
> > Thanks
> >
> > Scott
> >
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Friday, March 15, 2013 4:49 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: Lucene slow performance
> >
> > Hi,
> >
> > with standard configuartion, this cannot happen. What merge policy 
> > do you
> use? This looks to me like a misconfigured merge policy or using the 
> NoMergePolicy. With 3,000 segments, it will be slow, the question is, 
> why do you get those?
> >
> > Another thing could be: Do you always close IndexWriter after adding 
> > few
> documents and when closing, disable "wait for merge"? In that case, 
> all merges are interrupted and the merge policy never has a chance to 
> merge at all (because you are opening and closing IndexWriter all the 
> time with cancelling all merges)?
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >> -----Original Message-----
> >> From: Scott Smith [mailto:ssmith@mainstreamdata.com]
> >> Sent: Friday, March 15, 2013 11:15 PM
> >> To: java-user@lucene.apache.org
> >> Subject: Lucene slow performance
> >>
> >> We have a system that is using lucene and the searches are very slow.
> >> The number of documents is fairly small (less than 30,000) and each 
> >> document is typically only 2 to 10 kilo-characters.  Yet, searches 
> >> are taking
> 15-16 seconds.
> >>
> >> One of the things I noticed was that the index directory has 
> >> several thousand
> >> (3000+) .cfs files.  We do optimize the index once per day.  This 
> >> is a system that probably gets several thousand document deletes 
> >> and additions per day (spread out across the day).
> >>
> >> Any thoughts.  We didn't really notice this until we went to 4.x.
> >>
> >> Scott
> >>
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Mime
View raw message