Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4822B474.8020806@stimulussoft.com>
Date: Thu, 08 May 2008 10:06:12 +0200
From: Jamie <jamie@stimulussoft.com>
User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: Serious Index Corruption Error - FileNotFoundException
References: <126142c0805061426n1168421ya5594ef854fae5e4@mail.gmail.com>
 <B58D6ABE-CD90-4C8B-B05F-5C63BAFE9227@mikemccandless.com>
 <126142c0805061515p652410ebt1744c2ccd7afb8e2@mail.gmail.com>
 <0208CA7F-6822-4B1E-95B4-9BCF1ADD6EFB@mikemccandless.com>
 <48218C59.2030100@stimulussoft.com>
 <BCF0F18B-E6F8-4DEC-8B09-414C0202AD3B@mikemccandless.com>
 <4822B00F.3070308@stimulussoft.com>
 <E206CA64-F840-4E6D-994D-DF33FE3C5AAA@mikemccandless.com>
In-Reply-To: <E206CA64-F840-4E6D-994D-DF33FE3C5AAA@mikemccandless.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Michael

I had in fact preempted you and moved the delete lock code to a startup 
function. However, I found a nice little optimization that seems to 
force the writer to close when the process is manually killed. I added a 
JVM shutdown hook (i.e. using Runtime.getRuntime().addShutdownHook(this) 
) that closes the writer. This seems to reduce the number of instances 
where the lock file is left lying around.  Perhaps, it is something that 
could be included in the base lucene code. Also, are you aware of any 
chkdsk like utility for Lucene? i.e. in the event that an index is 
corrupted through disk error or otherwise?

Many thanks!

Jamie


Michael McCandless wrote:
>
> OK, that sounds like a legitimate reason to forcibly remove the write 
> lock, but it would be better to do that only on startup of your 
> process rather than in every openIndex() call.
>
> If ever you hit LockObtainFailedException in openIndex, even after 
> having deleted the write lock on startup, then that means there's a 
> bug (ie, two writers are in fact trying to open on the same index, one 
> of the two cases below).
>
> Mike
>
> Jamie wrote:
>> Hi Mike
>>
>> Thanks for the suggestions. I've implemented all of them. The main 
>> reason why I manually deleted the lock file was that sometimes users 
>> kill the server process manually or there is a hard reboot without 
>> any warning. In such circumstances, Lucene leaves a lock file lying 
>> around as it was busy writing to the index. Now, I understand that 
>> one shouldn't simply delete the lock file, but what do you suggest my 
>> users do? The server must continue running... the only way that I see 
>> how is to delete the lock file, unless there is the equivalent of 
>> chkdsk for Lucene indexes that I could run.
>>
>> Regards,
>>
>> Jmaie
>>
>>
>> Michael McCandless wrote:
>>>
>>> On quickly looking through the code I think there are some serious 
>>> hazards that could lead to this exception.
>>>
>>> First, in your openIndex code, if you hit a 
>>> LockObtainFailedException in trying to open your writer, you are 
>>> forcefully removing the write lock and then retrying.  Yet, you also 
>>> open an IndexReader to delete documents, which acquires the write 
>>> lock.  If ever you have this IndexReader open, and then you 
>>> forcefully remove the write lock and open the writer, that would 
>>> cause this exception.
>>>
>>> Second, you have a deletIndex method, which first tries to use the 
>>> writer with create=true (good) but then falls back to manually 
>>> removing the files.  Why is that fallback necessary?  If, for 
>>> example, you are also hitting a LockObtainFailedException, then 
>>> forcefully removing files while an IndexReader or IndexWriter holds 
>>> the write lock is also dangerous and would lead to this exception.
>>>
>>> In general it's very dangerous to forcibly remove, or ignore, 
>>> Lucene's write lock.  It really should only be necessary when 
>>> something catastrophic occurred (JVM crashed).
>>>
>>> Also, note that IndexWriter can now delete documents.  This would 
>>> simplify your code and possibly fix these two hazards.
>>>
>>> Do you see any of the error/warnings that you send to your logger?  
>>> (They would be corroborating evidence here).
>>>
>>> Mike


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org