lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-4462) Publishing flushed segments is single threaded and too costly
Date Mon, 08 Oct 2012 08:44:03 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Simon Willnauer updated LUCENE-4462:
------------------------------------

    Attachment: LUCENE-4462.patch

here is a patch that basically moves the prepareFlushedSegment into DWPT and calls it once
we are done flushing the segment. The publish call doesn't do any IO anymore which is good
and should not be a bottleneck. I could imagine that if somebody uses CFS this could be a
perf win too.

                
> Publishing flushed segments is single threaded and too costly
> -------------------------------------------------------------
>
>                 Key: LUCENE-4462
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4462
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>            Assignee: Simon Willnauer
>         Attachments: LUCENE-4462.patch
>
>
> Spinoff from http://lucene.markmail.org/thread/4li6bbomru35qn7w
> The new TestBagOfPostings failed the build because it timed out after 2 hours ... but
in digging I found that it was a starvation issue: the 4 threads were flushing segments much
faster than the 1 thread could publish them.
> I think this is because publishing segments (DocumentsWriter.publishFlushedSegment) is
actually rather costly (creates CFS file if necessary, writes .si, etc.).
> I committed a workaround for now, to prevent starvation (see svn diff -c 1394704 https://svn.apache.org/repos/asf/lucene/dev/trunk),
but we really should address the root cause by moving these costly ops into flush() so that
publishing is a low cost operation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message