lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: [jira] Commented: (LUCENE-743) IndexReader.reopen()
Date Tue, 13 Nov 2007 11:42:08 GMT
I was just trying to figure out why reopen() could fail ???

It would seem that on unix/linux the files could always be read  
(since they don't delete until last close), and on Windows they would  
be there (since they can't be deleted because they are open).

When "deleting segments", it would seem that the deleter should get  
an exclusive open on segments.xxx file first, before deleting any of  
the segments it refers to. If it does that, then it does seem that an  
open/reopen can ever fail mid-open. The only thing need is during an  
open/reopen is to back-off timeout if a read level shared lock cannot  
be obtained on the segments.xxx file.

For us it doesn't make much difference (no multi-process readers), so  
if everyone else is ok with the current behavior, we can move on...

Sorry if I wasted anyone's time.


On Nov 13, 2007, at 4:36 AM, Michael McCandless wrote:

>
> I've become lost in this thread.
>
> It started with the unit test in the reopen patch, suddenly jumped to
> NFS and lockless commits, then sync() got pulled in as well.
>
> The retry logic (part of lockless commits) already handles this case
> of a segments file being partially written & falling back to the
> previous one.
>
> Robert what problem are you trying to solve here?
>
> Mike
>
> "robert engels" <rengels@ix.netcom.com> wrote:
>> True. It seems that the Lucene code might be a bit more resilient
>> here though, using the following:
>>
>> 1. open the segments file exclusively (if this fails, updates are
>> prohibited, and an exception is thrown)
>> 2. write new segments
>> 3. write segments.new including segments hash & sync
>> 4. update segments file including hash
>> 5. delete segments that you can
>>
>> Then if it crashes in step 4, it is easy to know segments is bad (out
>> of date) and use segments.new
>> If it crashes in steps 3, then segments.new is easily detected as
>> being corrupt (hash does not match), so you know segments is valid.
>>
>> if there are segments that cannot be deleted in 5, every open can
>> check if it can delete them...
>>
>> A similar technique can be used if using lockless commits, just need
>> to make it segments.XXX.new, etc.
>>
>> On Nov 12, 2007, at 7:21 PM, Yonik Seeley wrote:
>>
>>> On Nov 12, 2007 7:19 PM, robert engels <rengels@ix.netcom.com>  
>>> wrote:
>>>> I would still argue that it is an incorrect setup - almost as  
>>>> bad as
>>>> "not plugging the computer in".
>>>
>>> A user themselves could even go in and look at the index files (I've
>>> done so myself)... as could a backup program or whatever.  It's a  
>>> fact
>>> of life on windows that a move or delete can fail.
>>>
>>> -Yonik
>>>
>>> -------------------------------------------------------------------- 
>>> -
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message