lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8310) Relax IWs check on pending deletes
Date Tue, 15 May 2018 21:50:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476524#comment-16476524
] 

Michael McCandless commented on LUCENE-8310:
--------------------------------------------

+1

> Relax IWs check on pending deletes
> ----------------------------------
>
>                 Key: LUCENE-8310
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8310
>             Project: Lucene - Core
>          Issue Type: Improvement
>    Affects Versions: 7.4, master (8.0)
>            Reporter: Simon Willnauer
>            Priority: Major
>             Fix For: 7.4, master (8.0)
>
>         Attachments: LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch, LUCENE-8310.patch
>
>
> I recently fixed the check in IW to fail if there are pending deletes. After upgrading
to a snapshot I realized the consequences of this check. It made most of our usage of IW to
for instance prepare commit metadata, rollback to safe commit-points etc. impossible since
we have to now busy wait on top of directory util all deletes are actually gone even though
that we can guarantee that our history always goes forward. ie we are truly append-only in
the sense of never reusing segment generations. The fix that I made was basically return false
from a _checkPendingDeletions_ in a directory wrapper to work around it.
> I do expect this to happen to a lot of lucene users even if they use IW correctly. My
proposal is to make the check in IW a bit more sophisticated and only fail if there are pending
deletes that are in the future from a generation perspective. We really don't care about files
from the past. My patch checks the segment generation of each pending file which is safe since
that is the same procedure we apply in IndexFileDeleter to inc reference etc. and only fail
if the pending delete is in the future.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message