lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <>
Subject [jira] [Updated] (LUCENE-3090) DWFlushControl does not take active DWPT out of the loop on fullFlush
Date Mon, 16 May 2011 08:09:52 GMT


Simon Willnauer updated LUCENE-3090:

    Attachment: LUCENE-3090.patch

next iteration. This patch changes the stalling mechanism from using num DWPT flushing to
netBytes and stalls at 2 x maxRamBuffer if we flush on num bytes. Stalling on memory consumption
allows to add an assert / upper bound to the netMemory which is nice but it doesn't help if
we are not flushing on RAM usage.

I think what we need to do (sep. issue) is to allow people to add a maxTotalUsedRam which
defaults to 2x maxRamBuffer if set or Runtime#maxMemory() / 2 if flushing by docCount to allow
us to stall indexing threads iff we cross that border and there is at least one DWPT flushing
or pending. 

> DWFlushControl does not take active DWPT out of the loop on fullFlush
> ---------------------------------------------------------------------
>                 Key: LUCENE-3090
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 4.0
>            Reporter: Simon Willnauer
>            Assignee: Simon Willnauer
>            Priority: Critical
>             Fix For: 4.0
>         Attachments: LUCENE-3090.patch, LUCENE-3090.patch, LUCENE-3090.patch
> We have seen several OOM on TestNRTThreads and all of them are caused by DWFlushControl
missing DWPT that are set as flushPending but can't full due to a full flush going on. Yet
that means that those DWPT are filling up in the background while they should actually be
checked out and blocked until the full flush finishes. Even further we currently stall on
the maxNumThreadStates while we should stall on the num of active thread states. I will attach
a patch tomorrow.

This message is automatically generated by JIRA.
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message