lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4456) IndexWriter makes unrefed files, and MockDir cannot detect it
Date Thu, 04 Oct 2012 01:17:07 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469054#comment-13469054
] 

Robert Muir commented on LUCENE-4456:
-------------------------------------

This is in 4.x now: just to try give some summary of this insanity...

# Uwe's original bug (LUCENE-4455) caused mike to add a sizeInBytes test to TestRollingUpdates
# this test failed in jenkins overnight, because of some unreferenced files leftover from
indexwriter. so it was clear MockDirectoryWrapper's unrefed files check has not really been
working for some time! We now have a "test-the-tester" test for this so we know its working
in the future!
# when we turned that on, all kinds of test failures happened: some exception/crash cases
and what not, some real bugs like deleting non-index files. some fake test failures because
tests made multiple commits. other fake test failures because IW couldnt delete files since
MockDirectoryWrapper acts like windows (and some reader had them open).
# We fixed MockDirectoryWrapper more so that it handles the openDeletedFiles Windows case,
and fixed it to handle indexes with multiple commits. More fake test failures because of hard-to-reproduce
act-like-Windows corner cases like segments_N and segments.gen (its hard to reproduce the
write at exactly the same time as these are slurped).
# To try to make those more reproducible, I figured MockDir could sometimes "hang onto" its
IndexInputs a little bit longer so this is more likely to happen in tests, by very rarely
doing a sleep in its close(). But a side effect of this, is testThreadInterruptDeadLock would
sometimes interrupt this sleeping in close, causing an exception to be thrown in an IndexInput's
close(). So this test (not intended for this purpose, but nice side effect) finds bugs where
exception handling of close() for indexinputs is wrong, because of its unclosed files check.

And lots of pulling out hair and stuff in between.

Anyway it probably looks a lot worse than it is, because most of these failures are silly
bugs/test bugs/trying to get the test framework right.

I have two jenkins pointed at branch_4x now:
* http://sierranevada.servebeer.com/job/branch4x-beaster/ (Linux)
* http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ (Windows)

Combined these do 70 test-core builds per hour. So every day is like a month's worth of test
failures, which makes things look really unstable but its intentional
to try to hunt everything down at once instead of onesy-twosey for a long time :)

Lets give it a little time and see if it can flush anything out and then backport this stuff
to 4.0, beast a bit more, and respin.

                
> IndexWriter makes unrefed files, and MockDir cannot detect it
> -------------------------------------------------------------
>
>                 Key: LUCENE-4456
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4456
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Robert Muir
>            Assignee: Michael McCandless
>         Attachments: LUCENE-4456_mdw_patch.txt, LUCENE-4456.patch, LUCENE-4456.patch
>
>
> Because MockDir calls crash() before it checks for unreferenced files, deletes are no
longer allowed.
> this means the unreferenced files check is useless!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message