lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4544) possible bug in ConcurrentMergeScheduler.merge(IndexWriter)
Date Wed, 07 Nov 2012 16:01:15 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492446#comment-13492446
] 

Michael McCandless commented on LUCENE-4544:
--------------------------------------------

I think it needs more than cutting over to thread pool to clean it up :)

We've actually looked at using a thread pool (see LUCENE-2063) but it apparently wasn't straightforward
... if you can see a way that'd be nice :)

But I think we should do that under a separate issue ... leave this one focused on the off-by-one
on maxMergeCount.

                
> possible bug in ConcurrentMergeScheduler.merge(IndexWriter) 
> ------------------------------------------------------------
>
>                 Key: LUCENE-4544
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4544
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/other
>    Affects Versions: 5.0
>            Reporter: Radim Kolar
>            Assignee: Michael McCandless
>         Attachments: LUCENE-4544.patch
>
>
> from dev list:
> ¨i suspect that this code is broken. Lines 331 - 343 in org.apache.lucene.index.ConcurrentMergeScheduler.merge(IndexWriter)
> mergeThreadCount() are currently active merges, they can be at most maxThreadCount, maxMergeCount
is number of queued merges defaulted with maxThreadCount+2 and it can never be lower then
maxThreadCount, which means that condition in while can never become true.
>       synchronized(this) {
>         long startStallTime = 0;
>         while (mergeThreadCount() >= 1+maxMergeCount) {
>           startStallTime = System.currentTimeMillis();
>           if (verbose()) {
>             message("    too many merges; stalling...");
>           }
>           try {
>             wait();
>           } catch (InterruptedException ie) {
>             throw new ThreadInterruptedException(ie);
>           }
>         } 
> While confusing, I think the code is actually nearly correct... but I
> would love to find some simplifications of CMS's logic (it's really
> hairy).
> It turns out mergeThreadCount() is allowed to go higher than
> maxThreadCount; when this happens, Lucene pauses
> mergeThreadCount()-maxThreadCount of those merge threads, and resumes
> them once threads finish (see updateMergeThreads).  Ie, CMS will
> accept up to maxMergeCount merges (and launch threads for them), but
> will only allow maxThreadCount of those threads to be running at once.
> So what that while loop is doing is preventing more than
> maxMergeCount+1 threads from starting, and then pausing the incoming
> thread to slow down the rate of segment creation (since merging cannot
> keep up).
> But ... I think the 1+ is wrong ... it seems like it should just be
> mergeThreadCount() >= maxMergeCount().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message