lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3090) DWFlushControl does not take active DWPT out of the loop on fullFlush
Date Fri, 13 May 2011 15:29:47 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033080#comment-13033080
] 

Simon Willnauer commented on LUCENE-3090:
-----------------------------------------

bq. But shouldn't stallControl kick in in that case? Ie, we stall all indexing if the number
of flush-pending DWPTs is >= the number of active DWPTs, I think?
Right so lets say we have two active thread states:
1. thread 1 starts indexing (max ram is 16M) it indexes n docs and has 15.9 MB ram used. Now
n+1 doc comes in has 5MB (active mem= 20.9M flush Mem: 0M)

2. take it out for flush (active mem=0M flush Mem: 20.9M)

3. thread 2 starts indexing and fills ram quickly ending up with 18M memory (active mem=18M
flush Mem: 20.9M)
4. take thread 2 out for flush (active mem=0M flush Mem: 38.9M)
5. thread 3 has already started indexing and reaches the RAM threshold (16M) so we have: (active
mem=16M flush Mem: 38.9M)
6. take it out for flushing (now we stall currently) (active mem=0M flush Mem: 54.9M) - this
is more than 3x max ram buffer.

we currently stall at  flush-pending DWPTs is > (num active DWPT + 1) we can reduce that
though but maybe we should swap back to ram based stalling?




> DWFlushControl does not take active DWPT out of the loop on fullFlush
> ---------------------------------------------------------------------
>
>                 Key: LUCENE-3090
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3090
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 4.0
>            Reporter: Simon Willnauer
>            Assignee: Simon Willnauer
>            Priority: Critical
>             Fix For: 4.0
>
>         Attachments: LUCENE-3090.patch, LUCENE-3090.patch
>
>
> We have seen several OOM on TestNRTThreads and all of them are caused by DWFlushControl
missing DWPT that are set as flushPending but can't full due to a full flush going on. Yet
that means that those DWPT are filling up in the background while they should actually be
checked out and blocked until the full flush finishes. Even further we currently stall on
the maxNumThreadStates while we should stall on the num of active thread states. I will attach
a patch tomorrow.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message