lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Lucene slow performance
Date Fri, 15 Mar 2013 23:29:21 GMT
OK, your configuration seems fine. I would have the following idea:
- Are you using windows? If yes, then IndexWriter cannot remove unused files when they are
still in use (e.g. hold by an open IndexReader)
- When you get a new IndexReader after changes to the index, do you close the old ones? If
not, the above will prevent IndexWriter from removing older cfs files. They are no longer
used, but linger around in filesystem. Because The older IndexReader stay open forever (if
you missed to close them), IndexWriter troies serveral times to delete them, but never succeeds.
On Unix/Linux, open files can be deleted, not on windows.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Simon Willnauer [mailto:simon.willnauer@gmail.com]
> Sent: Saturday, March 16, 2013 12:08 AM
> To: java-user@lucene.apache.org
> Subject: Re: Lucene slow performance
> 
> On Sat, Mar 16, 2013 at 12:02 AM, Scott Smith
> <ssmith@mainstreamdata.com> wrote:
> > " Do you always close IndexWriter after adding few documents and when
> closing, disable "wait for merge"? In that case, all merges are interrupted and
> the merge policy never has a chance to merge at all (because you are
> opening and closing IndexWriter all the time with cancelling all merges)?"
> >
> > Frankly I don't quite understand what this means.  When I "close" the
> indexwriter, I simply call close().  Is that the wrong thing?
> that should be fine...
> 
> this sounds very odd though, do you see file that get actually removed /
> merged if you call IndexWriter#forceMerge(1)
> 
> simon
> >
> > Thanks
> >
> > Scott
> >
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Friday, March 15, 2013 4:49 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: Lucene slow performance
> >
> > Hi,
> >
> > with standard configuartion, this cannot happen. What merge policy do you
> use? This looks to me like a misconfigured merge policy or using the
> NoMergePolicy. With 3,000 segments, it will be slow, the question is, why do
> you get those?
> >
> > Another thing could be: Do you always close IndexWriter after adding few
> documents and when closing, disable "wait for merge"? In that case, all
> merges are interrupted and the merge policy never has a chance to merge at
> all (because you are opening and closing IndexWriter all the time with
> cancelling all merges)?
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >> -----Original Message-----
> >> From: Scott Smith [mailto:ssmith@mainstreamdata.com]
> >> Sent: Friday, March 15, 2013 11:15 PM
> >> To: java-user@lucene.apache.org
> >> Subject: Lucene slow performance
> >>
> >> We have a system that is using lucene and the searches are very slow.
> >> The number of documents is fairly small (less than 30,000) and each
> >> document is typically only 2 to 10 kilo-characters.  Yet, searches are taking
> 15-16 seconds.
> >>
> >> One of the things I noticed was that the index directory has several
> >> thousand
> >> (3000+) .cfs files.  We do optimize the index once per day.  This is
> >> a system that probably gets several thousand document deletes and
> >> additions per day (spread out across the day).
> >>
> >> Any thoughts.  We didn't really notice this until we went to 4.x.
> >>
> >> Scott
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message