lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Lucene cluster with NFS or synchronization tool such as rsync
Date Mon, 04 Jul 2016 09:57:17 GMT
NFS is dangerous if different nodes may take turns writing to the shared
index.

Locking sometimes doesn't work correctly, client-side metadata caching
(e.g. the directory entry) can cause problems, NFS doesn't support "delete
on final close" semantics that Lucene relies on.

rsync-like behavior can work with IndexWriter if you use
SnapshotDeletionPolicy to hold a point-in-time view of the index open for
copying ... this is also how to take a live backup of a still-writing
index, and it's how Lucene's replication module works.

You could also try the new near-real-time replication, which copies just
the newly written segment files without requiring a full commit (fsync) on
the source index.

Mike McCandless

http://blog.mikemccandless.com

On Sun, Jul 3, 2016 at 2:09 PM, Desteny Child <myshareit@gmail.com> wrote:

> I need to organize a cluster for my stateless application based on Lucene
> 5.2.1. Right now I'm looking for a solution in order to share Lucene index
> via NFS or rsync between different Lucene nodes.
>
> Is it a good idea to use NFS for this purpose and if so will it be possible
> to read/write from different nodes to the same shared index ?
>
> Also I read that rsync tool can be used for this purpose(in order to
> synchronize index files across all nodes) but I can't find any success
> story for using rsync + Lucene. Right now I have a lot of question, one of
> them - is it safe to use rsync at anytime especially when IndexWriter is in
> progress(not closed) and actively indexes documents.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message