Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Fri, 13 May 2011 15:29:47 +0000 (UTC)
From: "Simon Willnauer (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <532012937.10330.1305300587684.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <945164257.7265.1305219347392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (LUCENE-3090) DWFlushControl does not take
 active DWPT out of the loop on fullFlush
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033080#comment-13033080 ] 

Simon Willnauer commented on LUCENE-3090:
-----------------------------------------

bq. But shouldn't stallControl kick in in that case? Ie, we stall all indexing if the number of flush-pending DWPTs is >= the number of active DWPTs, I think?
Right so lets say we have two active thread states:
1. thread 1 starts indexing (max ram is 16M) it indexes n docs and has 15.9 MB ram used. Now n+1 doc comes in has 5MB (active mem= 20.9M flush Mem: 0M)

2. take it out for flush (active mem=0M flush Mem: 20.9M)

3. thread 2 starts indexing and fills ram quickly ending up with 18M memory (active mem=18M flush Mem: 20.9M)
4. take thread 2 out for flush (active mem=0M flush Mem: 38.9M)
5. thread 3 has already started indexing and reaches the RAM threshold (16M) so we have: (active mem=16M flush Mem: 38.9M)
6. take it out for flushing (now we stall currently) (active mem=0M flush Mem: 54.9M) - this is more than 3x max ram buffer.

we currently stall at  flush-pending DWPTs is > (num active DWPT + 1) we can reduce that though but maybe we should swap back to ram based stalling?


> DWFlushControl does not take active DWPT out of the loop on fullFlush
> ---------------------------------------------------------------------
>
>                 Key: LUCENE-3090
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3090
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 4.0
>            Reporter: Simon Willnauer
>            Assignee: Simon Willnauer
>            Priority: Critical
>             Fix For: 4.0
>
>         Attachments: LUCENE-3090.patch, LUCENE-3090.patch
>
>
> We have seen several OOM on TestNRTThreads and all of them are caused by DWFlushControl missing DWPT that are set as flushPending but can't full due to a full flush going on. Yet that means that those DWPT are filling up in the background while they should actually be checked out and blocked until the full flush finishes. Even further we currently stall on the maxNumThreadStates while we should stall on the num of active thread states. I will attach a patch tomorrow.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org