lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <>
Subject Re: [jira] Commented: (LUCENE-743) IndexReader.reopen()
Date Mon, 12 Nov 2007 22:08:13 GMT
Then how can the commit during reopen be an issue?

I am not very family with this new code, but it seems that you need  
to write and then rename to segments.XXX.

As long as the files are sync'd, even on nfs the reopen should not  
see segments.XXX until is is ready.

Although lockless commits are beneficial in their own rite, I still  
think that people's understanding of NFS limitations are flawed. Read  
the section below on "close to open" consistency. There should be no  
problem using Lucene across NFS - even the old version.

The write-once nature of Lucene makes this trivial.  The only problem  
was the segments file, which is lucene used the read/write lock and  
close(0 correctly never would have been a problem.

According to the NFS docs:

NFS Version 2 requires that a server must save all the data in a  
write operation to disk before it replies to a client that the write  
operation has completed. This can be expensive because it breaks  
write requests into small chunks (8KB or less) that must each be  
written to disk before the next chunk can be written. Disks work best  
when they can write large amounts of data all at once.

NFS Version 3 introduces the concept of "safe asynchronous writes." A  
Version 3 client can specify that the server is allowed to reply  
before it has saved the requested data to disk, permitting the server  
to gather small NFS write operations into a single efficient disk  
write operation. A Version 3 client can also specify that the data  
must be written to disk before the server replies, just like a  
Version 2 write. The client specifies the type of write by setting  
the stable_how field in the arguments of each write operation to  
UNSTABLE to request a safe asynchronous write, and FILE_SYNC for an  
NFS Version 2 style write.

Servers indicate whether the requested data is permanently stored by  
setting a corresponding field in the response to each NFS write  
operation. A server can respond to an UNSTABLE write request with an  
UNSTABLE reply or a FILE_SYNC reply, depending on whether or not the  
requested data resides on permanent storage yet. An NFS protocol- 
compliant server must respond to a FILE_SYNC request only with a  
FILE_SYNC reply.

Clients ensure that data that was written using a safe asynchronous  
write has been written onto permanent storage using a new operation  
available in Version 3 called a COMMIT. Servers do not send a  
response to a COMMIT operation until all data specified in the  
request has been written to permanent storage. NFS Version 3 clients  
must protect buffered data that has been written using a safe  
asynchronous write but not yet committed. If a server reboots before  
a client has sent an appropriate COMMIT, the server can reply to the  
eventual COMMIT request in a way that forces the client to resend the  
original write operation. Version 3 clients use COMMIT operations  
when flushing safe asynchronous writes to the server during a close 
(2) or fsync(2) system call, or when encountering memory pressure.

A8. What is close-to-open cache consistency?
A. Perfect cache coherency among disparate NFS clients is very  
expensive to achieve, so NFS settles for something weaker that  
satisfies the requirements of most everyday types of file sharing.  
Everyday file sharing is most often completely sequential: first  
client A opens a file, writes something to it, then closes it; then  
client B opens the same file, and reads the changes.

So, when an application opens a file stored in NFS, the NFS client  
checks that it still exists on the server, and is permitted to the  
opener, by sending a GETATTR or ACCESS operation. When the  
application closes the file, the NFS client writes back any pending  
changes to the file so that the next opener can view the changes.  
This also gives the NFS client an opportunity to report any server  
write errors to the application via the return code from close().  
This behavior is referred to as close-to-open cache consistency.

Linux implements close-to-open cache consistency by comparing the  
results of a GETATTR operation done just after the file is closed to  
the results of a GETATTR operation done when the file is next opened.  
If the results are the same, the client will assume its data cache is  
still valid; otherwise, the cache is purged.

Close-to-open cache consistency was introduced to the Linux NFS  
client in 2.4.20. If for some reason you have applications that  
depend on the old behavior, you can disable close-to-open support by  
using the "nocto" mount option.

There are still opportunities for a client's data cache to contain  
stale data. The NFS version 3 protocol introduced "weak cache  
consistency" (also known as WCC) which provides a way of checking a  
file's attributes before and after an operation to allow a client to  
identify changes that could have been made by other clients.  
Unfortunately when a client is using many concurrent operations that  
update the same file at the same time, it is impossible to tell  
whether it was that client's updates or some other client's updates  
that changed the file.

For this reason, some versions of the Linux 2.6 NFS client abandon  
WCC checking entirely, and simply trust their own data cache. On  
these versions, the client can maintain a cache full of stale file  
data if a file is opened for write. In this case, using file locking  
is the best way to ensure that all clients see the latest version of  
a file's data.

A system administrator can try using the "noac" mount option to  
achieve attribute cache coherency among multiple clients. Almost  
every client operation checks file attribute information. Usually the  
client keeps this information cached for a period of time to reduce  
network and server load. When "noac" is in effect, a client's file  
attribute cache is disabled, so each operation that needs to check a  
file's attributes is forced to go back to the server. This permits a  
client to see changes to a file very quickly, at the cost of many  
extra network operations.

Be careful not to confuse "noac" with "no data caching." The "noac"  
mount option will keep file attributes up-to-date with the server,  
but there are still races that may result in data incoherency between  
client and server. If you need absolute cache coherency among  
clients, applications can use file locking, where a client purges  
file data when a file is locked, and flushes changes back to the  
server before unlocking a file; or applications can open their files  
with the O_DIRECT flag to disable data caching entirely.

For a better understanding of the compromises faced in the design of  
NFS caching, see Callaghan's "NFS Illustrated."

On Nov 12, 2007, at 3:47 PM, Yonik Seeley wrote:

> On Nov 12, 2007 4:43 PM, robert engels <> wrote:
>> Why doesn't reopen get the 'read' lock, since commit has the write
>> lock, it should wait...
> After lockless commits, there is no read lock!
> -Yonik
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message