lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <simon.willna...@googlemail.com>
Subject Re: Thread locking while merging (ConcurrentMergeScheduler issue?)
Date Wed, 03 Aug 2011 21:45:24 GMT
On Wed, Aug 3, 2011 at 7:22 PM, Pierre-Henri Toussaint
<pierrehenri.toussaint@gmail.com> wrote:
> Hello,
>
> First many thanks for getting in touch.
>
>
> Michael McCandless-2 wrote:
>>
>> It looks like merging is running too slowly in your environment,
>> relative to indexing; all of your indexing threads are stuck wanting
>> to launch a new merge but there's already the max allowed (1)
>> concurrent merge running and so IW (intentionally) stalls them.
>>
>>>> It's exactly what I just noticed. And 5 indexing thread vs 1 merge
>>>> doesn't sound fair.
> I read ConcurrentMergeScheduler source code and found out that if
> maxMergeCount is reached, incoming threads are forcefully paused, which
> totally make sense. I should try to raise a little bit the maxMergeCount to
> be accorded with indexing.
>
> However, I still don't understand why the merging take so much time (few
> minutes to an hour). During that time, the IO are almost non existant and
> CPU stays low. That's why I suspected an locking issue.
>
>
> Michael McCandless-2 wrote:
>>
>> Are you sure you passed 2 for numThreads to ThreadedIndexWriter?  I see 5
>> threads in your stack traces, I think.
>>>> I actually passed 5 threads in ThreadedIndexWriter, my mistake. Post
>>>> edited.
>
>
> Michael McCandless-2 wrote:
>>
>> What kind of IO system do you have?
>>>> slow macbook laptop harddrive. I'm actually just trying to get my grips
>>>> with this wonderful library before, I hope, being able to use it on a
>>>> proper system.
>
>
> Michael McCandless-2 wrote:
>>
>> Can you setInfoStream in the IW and post the output?  Could give more
>> clues...
>>>> will post that after my next test
>
>
> Simon Willnauer wrote:
>>
>> can you explain your configuration as well as the problem? I see one
>> thread merging (Runnable state) and three other merge threads that are
>> waiting on a lock. it seems that CMS has put them into wait state and
>> your indexing threads are blocking on a stall control (too many
>> merging threads). is it possible that you are flushing very often ie.
>> low ram buffer or low maxBufferedDocs so that many segments are
>> created?
>>
>>>> I think the flushing is fine (RAMBufferSizeMB:256mb)
> My configuration is a macbookpro 2.4ghz/4GBram. Not ideal, but the goal is
> not to benchmark Lucene.

I rather refer to your IndexWriterConfig here, can you post it and
maybe tell me a little about how you index, what you index etc. how
many segments do you have, if you optimize etc. not sure why this is
happening but maybe something that looks unrelated to you can help?

simon
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Thread-locking-while-merging-ConcurrentMergeScheduler-issue-tp3222427p3222946.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message