lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre-Henri Toussaint <>
Subject Re: Thread locking while merging (ConcurrentMergeScheduler issue?)
Date Wed, 03 Aug 2011 17:22:17 GMT

First many thanks for getting in touch.

Michael McCandless-2 wrote:
> It looks like merging is running too slowly in your environment,
> relative to indexing; all of your indexing threads are stuck wanting
> to launch a new merge but there's already the max allowed (1)
> concurrent merge running and so IW (intentionally) stalls them.
>>> It's exactly what I just noticed. And 5 indexing thread vs 1 merge
>>> doesn't sound fair.
I read ConcurrentMergeScheduler source code and found out that if
maxMergeCount is reached, incoming threads are forcefully paused, which
totally make sense. I should try to raise a little bit the maxMergeCount to
be accorded with indexing.

However, I still don't understand why the merging take so much time (few
minutes to an hour). During that time, the IO are almost non existant and
CPU stays low. That's why I suspected an locking issue.

Michael McCandless-2 wrote:
> Are you sure you passed 2 for numThreads to ThreadedIndexWriter?  I see 5
> threads in your stack traces, I think.
>>> I actually passed 5 threads in ThreadedIndexWriter, my mistake. Post
>>> edited.

Michael McCandless-2 wrote:
> What kind of IO system do you have?
>>> slow macbook laptop harddrive. I'm actually just trying to get my grips
>>> with this wonderful library before, I hope, being able to use it on a
>>> proper system.

Michael McCandless-2 wrote:
> Can you setInfoStream in the IW and post the output?  Could give more
> clues...
>>> will post that after my next test

Simon Willnauer wrote:
> can you explain your configuration as well as the problem? I see one 
> thread merging (Runnable state) and three other merge threads that are 
> waiting on a lock. it seems that CMS has put them into wait state and 
> your indexing threads are blocking on a stall control (too many 
> merging threads). is it possible that you are flushing very often ie. 
> low ram buffer or low maxBufferedDocs so that many segments are 
> created?
>>> I think the flushing is fine (RAMBufferSizeMB:256mb)
My configuration is a macbookpro 2.4ghz/4GBram. Not ideal, but the goal is
not to benchmark Lucene.

View this message in context:
Sent from the Lucene - Java Users mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message