Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=dk20050327; d=ix.netcom.com;
  b=UTyQl8VXtYi9sTyEu/ZvbslwQMWI/ziNM3sYM14O37JujnvJNFsTpuCjgEbgFxxQ;
  h=Received:Mime-Version:In-Reply-To:References:Content-Type:Message-Id:Content-Transfer-Encoding:From:Subject:Date:To:X-Mailer:X-ELNK-Trace:X-Originating-IP;
Mime-Version: 1.0 (Apple Message framework v752.3)
In-Reply-To: <1194908079.21750.1220937951@webmail.messagingengine.com>
References: <11500990.1194903350969.JavaMail.jira@brutus>
 <F466D809-08AD-4A55-85F4-E01B02263ABC@ix.netcom.com>
 <c68e39170711121347k7a08a9fv3d6d9c423678311@mail.gmail.com>
 <BFD76B8A-0726-4A53-9A1A-2897CC7702A4@ix.netcom.com>
 <1194908079.21750.1220937951@webmail.messagingengine.com>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <F4D3B4CD-D62C-4B93-86C7-C1F83CE66068@ix.netcom.com>
Content-Transfer-Encoding: 7bit
From: robert engels <rengels@ix.netcom.com>
Subject: Re: [jira] Commented: (LUCENE-743) IndexReader.reopen()
Date: Mon, 12 Nov 2007 17:06:26 -0600
To: java-dev@lucene.apache.org

What are you basing the "rename" is not reliable on windows on? That  
a virus scanner has the file open. If that is the case, that should  
either be an incorrect setup, or the operation retried until it  
completes.

Writing directly to a file that someone else can open for reading is  
bound to be a problem. If the file is opened exclusive for write,  
then the others will be prohibited from opening for read, so there  
should not be a problem.

All of the "delete on last close" stuff is a poor design. The  
database can be resync on startup.

The basic design flaw is one I have pointed out many times - you  
either use Lucene in a local environment, or a server environment.  
Using NFS to "share" a Lucene database is a poor choice (normally due  
to performance, but there are other problems - e.g. resource and user  
monitoring, etc.) is a poor choice !.

People have written reliable database systems without very advanced  
semantics for years. There is no reason for all of this esoteric code  
in Lucene.

Those that claim, Lucene had problems with NFS in the past, did not  
perform reliable testing, or their OS was out of date.  What is  
Lucene was failing for an OS needed an update, would you change  
Lucene, or fix/update the OS??? Obviously the former.

Some very loud voices complained about the NFS problems without doing  
the due diligence and test cases to prove the problem. Instead they  
just mucked up the Lucene code.


On Nov 12, 2007, at 4:54 PM, Michael McCandless wrote:

>
> robert engels <rengels@ix.netcom.com> wrote:
>
>> Then how can the commit during reopen be an issue?
>
> This is what happens:
>
>   * Reader opens latest segments_N & reads all SegmentInfos
>     successfully.
>
>   * Writer writes new segments_N+1, and then deletes now un-referenced
>     files.
>
>   * Reader tries to open files referenced by segments_N and hits FNFE
>     when it tries to open a file writer just removed.
>
> Lucene handles this fine (it just retries on the new segments_N+1),
> but the patch in LUCENE-743 is now failing to decRef the Norm
> instances when this retry happens.
>
>> I am not very family with this new code, but it seems that you need
>> to write segments.XXX.new and then rename to segments.XXX.
>
> We don't rename anymore (it's not reliable on windows).  We write
> straight to segments_N.
>
>> As long as the files are sync'd, even on nfs the reopen should not
>> see segments.XXX until is is ready.
>>
>> Although lockless commits are beneficial in their own rite, I still
>> think that people's understanding of NFS limitations are
>> flawed. Read the section below on "close to open" consistency. There
>> should be no problem using Lucene across NFS - even the old version.
>>
>> The write-once nature of Lucene makes this trivial.  The only
>> problem was the segments file, which is lucene used the read/write
>> lock and close(0 correctly never would have been a problem.
>
> Yes, in an ideal world, NFS server+clients are supposed to implement
> close-to-open semantics but in my experience they do not always
> succeed.  Previous version of Lucene do in fact have problems over
> NFS.  NFS also does not give you "delete on last close" which Lucene
> normally relies on (unless you create a custom deletion policy).
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org