lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Serious Index Corruption Error - FileNotFoundException
Date Thu, 08 May 2008 08:38:47 GMT

It would make me nervous to have Lucene insert that shutdown hook.   
EG closing the IndexWriter could in general be a time-consuming  
process.  But if it's working for you, that's great.  Though, if you  
explicitly kill the JVM (eg kill -9) those shutdown hooks won't run.

You should use org.apache.lucene.index.CheckIndex to check the health  
of your index.

Mike

On May 8, 2008, at 4:06 AM, Jamie wrote:
> Hi Michael
>
> I had in fact preempted you and moved the delete lock code to a  
> startup function. However, I found a nice little optimization that  
> seems to force the writer to close when the process is manually  
> killed. I added a JVM shutdown hook (i.e. using Runtime.getRuntime 
> ().addShutdownHook(this) ) that closes the writer. This seems to  
> reduce the number of instances where the lock file is left lying  
> around.  Perhaps, it is something that could be included in the  
> base lucene code. Also, are you aware of any chkdsk like utility  
> for Lucene? i.e. in the event that an index is corrupted through  
> disk error or otherwise?
>
> Many thanks!
>
> Jamie
>
>
> Michael McCandless wrote:
>>
>> OK, that sounds like a legitimate reason to forcibly remove the  
>> write lock, but it would be better to do that only on startup of  
>> your process rather than in every openIndex() call.
>>
>> If ever you hit LockObtainFailedException in openIndex, even after  
>> having deleted the write lock on startup, then that means there's  
>> a bug (ie, two writers are in fact trying to open on the same  
>> index, one of the two cases below).
>>
>> Mike
>>
>> Jamie wrote:
>>> Hi Mike
>>>
>>> Thanks for the suggestions. I've implemented all of them. The  
>>> main reason why I manually deleted the lock file was that  
>>> sometimes users kill the server process manually or there is a  
>>> hard reboot without any warning. In such circumstances, Lucene  
>>> leaves a lock file lying around as it was busy writing to the  
>>> index. Now, I understand that one shouldn't simply delete the  
>>> lock file, but what do you suggest my users do? The server must  
>>> continue running... the only way that I see how is to delete the  
>>> lock file, unless there is the equivalent of chkdsk for Lucene  
>>> indexes that I could run.
>>>
>>> Regards,
>>>
>>> Jmaie
>>>
>>>
>>> Michael McCandless wrote:
>>>>
>>>> On quickly looking through the code I think there are some  
>>>> serious hazards that could lead to this exception.
>>>>
>>>> First, in your openIndex code, if you hit a  
>>>> LockObtainFailedException in trying to open your writer, you are  
>>>> forcefully removing the write lock and then retrying.  Yet, you  
>>>> also open an IndexReader to delete documents, which acquires the  
>>>> write lock.  If ever you have this IndexReader open, and then  
>>>> you forcefully remove the write lock and open the writer, that  
>>>> would cause this exception.
>>>>
>>>> Second, you have a deletIndex method, which first tries to use  
>>>> the writer with create=true (good) but then falls back to  
>>>> manually removing the files.  Why is that fallback necessary?   
>>>> If, for example, you are also hitting a  
>>>> LockObtainFailedException, then forcefully removing files while  
>>>> an IndexReader or IndexWriter holds the write lock is also  
>>>> dangerous and would lead to this exception.
>>>>
>>>> In general it's very dangerous to forcibly remove, or ignore,  
>>>> Lucene's write lock.  It really should only be necessary when  
>>>> something catastrophic occurred (JVM crashed).
>>>>
>>>> Also, note that IndexWriter can now delete documents.  This  
>>>> would simplify your code and possibly fix these two hazards.
>>>>
>>>> Do you see any of the error/warnings that you send to your  
>>>> logger?  (They would be corroborating evidence here).
>>>>
>>>> Mike
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message