lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Thread locking while merging (ConcurrentMergeScheduler issue?)
Date Wed, 03 Aug 2011 21:58:09 GMT
On Wed, Aug 3, 2011 at 1:22 PM, Pierre-Henri Toussaint
<pierrehenri.toussaint@gmail.com> wrote:

>> It looks like merging is running too slowly in your environment,
>> relative to indexing; all of your indexing threads are stuck wanting
>> to launch a new merge but there's already the max allowed (1)
>> concurrent merge running and so IW (intentionally) stalls them.
>>
>>>> It's exactly what I just noticed. And 5 indexing thread vs 1 merge
>>>> doesn't sound fair.
> I read ConcurrentMergeScheduler source code and found out that if
> maxMergeCount is reached, incoming threads are forcefully paused, which
> totally make sense. I should try to raise a little bit the maxMergeCount to
> be accorded with indexing.

Well, the core problem is that merging is running too slowly vs how
quickly your indexing threads are generating new segments -- so if you
raise maxMergeCount, you'll only cause merging to fall even more
behind.

> However, I still don't understand why the merging take so much time (few
> minutes to an hour). During that time, the IO are almost non existant and
> CPU stays low. That's why I suspected an locking issue.

Yeah that's strange.  It could simply be you are using too many
indexing threads (plus laptop drive being slowish) -- since you only
have 2 cores (Core 2 Duo) maybe cut back from 5 to 2 threads?

>> What kind of IO system do you have?
>>>> slow macbook laptop harddrive. I'm actually just trying to get my grips
>>>> with this wonderful library before, I hope, being able to use it on a
>>>> proper system.

This could explain the slow merging.

>> Can you setInfoStream in the IW and post the output?  Could give more
>> clues...
>>>> will post that after my next test

OK maybe we'll get more insight...

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message