lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2637) FSDirectory.copyBytes isn't safe for SimpleFSDirectory
Date Thu, 09 Sep 2010 09:38:33 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907569#action_12907569
] 

Michael McCandless commented on LUCENE-2637:
--------------------------------------------

For the record, the problem w/ the opto is that it pulls the "unique" FileChannel from the
RandomAccessFile, positions it, and does non-positional reads.  This is not safe, because
SimpleFSDir also does non-positional reads.  So eg if a BG merge is running, invoking .copyBytes
when merging the fields, and at the same time another thread is eg loading a stored doc from
the same open file w/ SimpleFSDIr, they conflict because they are both relying on the stateful
position of RAF.

NIOFSDir always does positional reads so it doesn't hit this.

But I agree we should pull the opto from 3x & trunk, open new issue to re-add it, only
add it back to trunk once we have solid tests.  EG the new TestCopyBytes I added does not
catch this thread-safety issue (but I think, though I didn't test, should catch the original
corruption issue w/ the opto).

And, yes, we must now notify 3x/trunk users, who uses SimpleFSDir (because they are on windows
or because they explicitly instantiated it) that they may have latent index corruption and
should run CheckIndex.

> FSDirectory.copyBytes isn't safe for SimpleFSDirectory
> ------------------------------------------------------
>
>                 Key: LUCENE-2637
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2637
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 3.1, 4.0
>            Reporter: Robert Muir
>            Assignee: Robert Muir
>         Attachments: LUCENE-2637.patch, LUCENE-2637.patch
>
>
> the copyBytes optimization from LUCENE-2574 is not safe for SimpleFSDirectory, but works
fine for NIOFSDirectory.
> With SimpleFSDirectory, the copyBytes optimization causes index corruption.
> see http://www.lucidimagination.com/search/document/36d2dbfc691909d5/bug_triggered_by_testindexwriter_testrandomstoredfields
for background
> here are my steps to reproduce (most of the time, at least on windows):
> {noformat}
> 1. edit line 87 of TestIndexWriter to plugin the seed:
>     random = newRandom(3312389322103990899L);
> 2. edit line 5138 of TestIndexWriter to force SimpleFSDirectory:
>     Directory dir = new SimpleFSDirectory(index);
> 3. run this command:
>     ant clean test-core -Dtestcase=TestIndexWriter
> -Dtestmethod=testRandomStoredFields -Dtests.iter=10
> -Dtests.codec="MockVariableIntBlock(29)"
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message