lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <>
Subject Re: [jira] Commented: (LUCENE-665) temporary file access denied on Windows
Date Wed, 13 Sep 2006 19:40:39 GMT
This is a server application. Windows indexing service should be  
disable on the directories that contain Lucene locks and files.

This is the same procedure that would be required for any database.

On Sep 13, 2006, at 2:27 PM, Michael McCandless (JIRA) wrote:

>     [ 
> page=comments#action_12434527 ]
> Michael McCandless commented on LUCENE-665:
> -------------------------------------------
> I do think we should make Lucene robust to "windows change log"
> software.
> We could take the position that you have to uninstall such software
> because they "conflict" with Lucene, but I don't think that's
> realistic.  Apparently many packages use this convenient API and that
> will only get worse with time.
> I would put this under the "Lucene should assume the least common
> denominator of filesystem's capabilities" umbrella.  Meaning, Lucene
> now assumes it can rename files right after closing them, but on
> Windows this isn't a safe assumption so if possible we should change
> the index format to not require this.
> I will try to reproduce this bug with my [upcoming] changes for
> lockless commits (numbered segments files) -- the lockless commits
> changes do much less file renaming, so the issue should be rarer (but
> could still occur).
>> temporary file access denied on Windows
>> ---------------------------------------
>>                 Key: LUCENE-665
>>                 URL:
>>             Project: Lucene - Java
>>          Issue Type: Bug
>>          Components: Store
>>    Affects Versions: 2.0.0
>>         Environment: Windows
>>            Reporter: Doron Cohen
>>         Attachments: FSDirectory_Retry_Logic.patch,  
>> FSDirs_Retry_Logic_3.patch, Test_Output.txt,  
>> When interleaving adds and removes there is frequent opening/ 
>> closing of readers and writers.
>> I tried to measure performance in such a scenario (for issue 565),  
>> but the performance test failed  - the indexing process crashed  
>> consistently with file "access denied" errors - "cannot create a  
>> lock file" in "lockFile.createNewFile()" and "cannot rename file".
>> This is related to:
>> - issue 516 (a closed issue: "TestFSDirectory fails on Windows") -  
>> - user list questions due to file errors:
>>   - 
>> Denied-errors-tf1649795.html
>>   - 
>> windows-service-on-xp%2C-crashing-tf2053536.html
>> - discussion on lock-less commits 
>> commits-tf2126935.html
>> My test setup is: XP (SP1), JAVA 1.5 - both SUN and IBM SDKs.
>> I noticed that the problem is more frequent when locks are created  
>> on one disk and the index on another. Both are NTFS with Windows  
>> indexing service enabled. I suspect this indexing service might be  
>> related - keeping files busy for a while, but don't know for sure.
>> After experimenting with it I conclude that these problems - at  
>> least in my scenario - are due to a temporary situation - the FS,  
>> or the OS, is *temporarily* holding references to files or  
>> folders, preventing from renaming them, deleting them, or creating  
>> new files in certain directories.
>> So I added to FSDirectory a retry logic in cases the error was  
>> related to "Access Denied". This is the same approach brought in  
>> service-on-xp%2C-crashing-tf2053536.html - there, in addition to  
>> the retry, gc() is invoked (I did not gc()). This is based on the  
>> *hope* that a access-denied situation would vanish after a small  
>> delay, and the retry would succeed.
>> I modified FSDirectory this way for "Access Denied" errors during  
>> creating a new files, renaming a file.
>> This worked fine for me. The performance test that failed before,  
>> now managed to complete. There should be no performance  
>> implications due to this modification, because only the cases that  
>> would otherwise wrongly fail are now delaying some extra millis  
>> and retry.
>> I am attaching here a patch - FSDirectory_Retry_Logic.patch - that  
>> has these changes to FSDirectory.
>> All "ant test" tests pass with this patch.
>> Also attaching a test case that demostrates the problem - at least  
>> on my machine. There two tests cases in that test file - one that  
>> works in system temp (like most Lucene tests) and one that creates  
>> the index in a different disk. The latter case can only run if the  
>> path ("D:" , "tmp") is valid.
>> It would be great if people that experienced these problems could  
>> try out this patch and comment whether it made any difference for  
>> them.
>> If it turns out useful for others as well, including this patch in  
>> the code might help to relieve some of those "frustration" user  
>> cases.
>> A comment on state of proposed patch:
>> - It is not a "ready to deploy" code - it has some debug printing,  
>> showing the cases that the "retry logic" actually took place.
>> - I am not sure if current 30ms is the right delay... why not  
>> 50ms? 10ms? This is currently defined by a constant.
>> - Should a call to gc() be added? (I think not.)
>> - Should the retry be attempted also on "non access-denied"  
>> exceptions? (I think not).
>> - I feel it is somewhat "woodoo programming", but though I don't  
>> like it, it seems to work...
>> Attached files:
>> 1. - the LONG test that fails on  
>> XP without the patch and passes with the patch.
>> 2. FSDirectory_Retry_Logic.patch
>> 3. Test_Output.txt- output of the test with the patch, on my XP.  
>> Only the createNewFile() case had to be bypassed in this test, but  
>> for another program I also saw the renameFile() being bypassed.
>> - Doron
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators: 
> Administrators.jspa
> -
> For more information on JIRA, see: 
> software/jira
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message