Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Message-ID: <32454317.118061275435335975.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 19:35:35 -0400 (EDT)
From: "Michael McCandless (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Subject: [jira] Resolved: (LUCENE-2161) Some concurrency improvements for
 NRT
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/LUCENE-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-2161.
----------------------------------------

    Resolution: Fixed

> Some concurrency improvements for NRT
> -------------------------------------
>
>                 Key: LUCENE-2161
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2161
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.9.3, 3.0.2, 3.1, 4.0
>
>         Attachments: LUCENE-2161.patch
>
>
> Some concurrency improvements for NRT
> I found & fixed some silly thread bottlenecks that affect NRT:
>   * Multi/DirectoryReader.numDocs is synchronized, I think so only 1
>     thread computes numDocs if it's -1.  I removed this sync, and made
>     numDocs volatile, instead.  Yes, multiple threads may compute the
>     numDocs for the first time, but I think that's harmless?
>   * Fixed BitVector's ctor to set count to 0 on creating a new BV, and
>     clone to copy the count over; this saves CPU computing the count
>     unecessarily.
>   * Also strengthened assertions done in SR, testing the delete docs
>     count.
> I also found an annoying thread bottleneck that happens, due to CMS.
> Whenever CMS hits the max running merges (default changed from 3 to 1
> recently), and the merge policy now wants to launch another merge, it
> forces the incoming thread to wait until one of the BG threads
> finishes.
> This is a basic crude throttling mechanism -- you force the mutators
> (whoever is causing new segments to appear) to stop, so that merging
> can catch up.
> Unfortunately, when stressing NRT, that thread is the one that's
> opening a new NRT reader.
> So, the first serious problem happens when you call .reopen() on your
> NRT reader -- this call simply forwards to IW.getReader if the reader
> was an NRT reader.  But, because DirectoryReader.doReopen is
> synchronized, this had the horrible effect of holding the monitor lock
> on your main IR.  In my test, this blocked all searches (since each
> search uses incRef/decRef, still sync'd until LUCENE-2156, at least).
> I fixed this by making doReopen only sync'd on this if it's not simply
> forwarding to getWriter.  So that's a good step forward.
> This prevents searches from being blocked while trying to reopen to a
> new NRT.
> However... it doesn't fix the problem that when an immense merge is
> off and running, opening an NRT reader could hit a tremendous delay
> because CMS blocks it.  The BalancedSegmentMergePolicy should help
> here... by avoiding such immense merges.
> But, I think we should also pursue an improvement to CMS.  EG, if it
> has 2 merges running, where one is huge and one is tiny, it ought to
> increase thread priority of the tiny one.  I think with such a change
> we could increase the max thread count again, to prevent this
> starvation.  I'll open a separate issue....

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org