lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter A. Friend" <octav...@corp.earthlink.net>
Subject Re: NFS and Lucene 2.0 status - still troublesome ?
Date Mon, 13 Nov 2006 16:45:01 GMT

On Nov 13, 2006, at 8:10 AM, √ėyvind Stegard wrote:

> I've searched the list and have found many references to problems when
> using Lucene over NFS. Mostly because of file-based locking, which
> doesn't work all that well for many NFS installations. I'm under the
> impression that the core locking logic between writers and/or readers
> hasn't changed in a significant way between Lucene 1.4 and 2.0 (?). I
> guess this means NFS is still problematic ?

Unfortunately it all depends on the reliability of the NFS drivers in  
the OS, and the kind of filers you are using. If the environment  
isn't too busy, NFS lockd *may* work on some systems, but it usually  
ends up collapsing under load.

 From there you have to hand craft some C code to create lock files,  
and what works again depends on your system. On some systems doing an  
exclusive create will work (can only be expected to work on version 3  
mounts), but then local caches will bite you, so you end up having to  
disable the directory cache, assuming your system supports such an  
option. That failing, creating locks as symlinks to unique temporary  
files that don't exist will usually blow through the cache and work  
ok. This of course doesn't rule out problems in the NFS  
implementation that show up under heavy load, and allow more than one  
machine to think it has the lock. You also have to include some code  
to sensibly expire locks left from crashes.

> We are considering a model where a single node updates the search  
> index
> according to changes in the repository (only one physical index for  
> the
> entire cluster) while multiple other nodes can search the very same
> index over NFS (read-only). But I guess there is a need for a single
> lock-directory shared and writable between all nodes, and that this
> makes NFS-usage difficult ?

The fact that only a single node will be doing writes greatly  
improves the chances of this working. I don't recall whether readers  
ever check for locks, it's best if that can be avoided. I know that  
it's safe to write the new indexes since they aren't being referred  
to by the segments file, but I'm not sure what sequence of operations  
are used when re-writing the segments file. I think unlinking the old  
segments file and using a rename to put the new one in place is  
probably the safest bet.

Peter


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message