lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre-Henri Toussaint <pierrehenri.toussa...@gmail.com>
Subject Re: Thread locking while merging (ConcurrentMergeScheduler issue?)
Date Wed, 03 Aug 2011 17:22:17 GMT
Hello,

First many thanks for getting in touch.


Michael McCandless-2 wrote:
> 
> It looks like merging is running too slowly in your environment,
> relative to indexing; all of your indexing threads are stuck wanting
> to launch a new merge but there's already the max allowed (1)
> concurrent merge running and so IW (intentionally) stalls them.
> 
>>> It's exactly what I just noticed. And 5 indexing thread vs 1 merge
>>> doesn't sound fair.
I read ConcurrentMergeScheduler source code and found out that if
maxMergeCount is reached, incoming threads are forcefully paused, which
totally make sense. I should try to raise a little bit the maxMergeCount to
be accorded with indexing.

However, I still don't understand why the merging take so much time (few
minutes to an hour). During that time, the IO are almost non existant and
CPU stays low. That's why I suspected an locking issue.


Michael McCandless-2 wrote:
> 
> Are you sure you passed 2 for numThreads to ThreadedIndexWriter?  I see 5
> threads in your stack traces, I think.
>>> I actually passed 5 threads in ThreadedIndexWriter, my mistake. Post
>>> edited.


Michael McCandless-2 wrote:
> 
> What kind of IO system do you have?
>>> slow macbook laptop harddrive. I'm actually just trying to get my grips
>>> with this wonderful library before, I hope, being able to use it on a
>>> proper system.


Michael McCandless-2 wrote:
> 
> Can you setInfoStream in the IW and post the output?  Could give more
> clues...
>>> will post that after my next test


Simon Willnauer wrote:
> 
> can you explain your configuration as well as the problem? I see one 
> thread merging (Runnable state) and three other merge threads that are 
> waiting on a lock. it seems that CMS has put them into wait state and 
> your indexing threads are blocking on a stall control (too many 
> merging threads). is it possible that you are flushing very often ie. 
> low ram buffer or low maxBufferedDocs so that many segments are 
> created?
> 
>>> I think the flushing is fine (RAMBufferSizeMB:256mb)
My configuration is a macbookpro 2.4ghz/4GBram. Not ideal, but the goal is
not to benchmark Lucene.

--
View this message in context: http://lucene.472066.n3.nabble.com/Thread-locking-while-merging-ConcurrentMergeScheduler-issue-tp3222427p3222946.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message