lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Closing IndexWriter can be very slow on large indexes
Date Tue, 26 Jul 2011 16:30:47 GMT
Which method (abort or close) do you see taking so much time?

It's odd, because IW.abort should quickly stop any running BG merges.

Can you get a dump of the thread stacks during this long abort/close
and post that back?

Can't answer if Lucene 3.x will improve this situation until we find
the source of the slowness...

Mike McCandless

http://blog.mikemccandless.com

On Tue, Jul 26, 2011 at 11:33 AM, Chris Bamford
<chris.bamford@talktalk.net> wrote:
> Hi
>
> I think I must be doing something wrong, but not sure what.
>
> I have some long running indexing code which sometimes needs to be shutdown in a hurry.
 To achieve this, I set a shutdown flag which causes it to break from the loop and call first
abort() and then close().  The problem is that with a large index (say, 15Gb) in Lucene 2.3.2,
it can take over an hour.  (Yes, I know I should be on a later version of Lucene, but that's
another issue - we are stuck with this for now!).
>
> The IW is opened in autoCommit mode and mergeFactor=10.
>
> During this closedown stage, the indexes are being constantly updated by Lucene itself,
making me suspect it could be merging.
>
> Firstly, can someone explain what it is doing under the covers that takes so long? (And
any action I can take to get around it)
>
> Second, if I were to rebuild the code with say, Lucene 3 and run it in compatibility
mode with the 2.3.2 indexes, would I have a richer set of tools I could use to overcome the
issue?
>
> Thanks,
>
> - Chris
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message