lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Lucene cluster with NFS or synchronization tool such as rsync
Date Thu, 07 Jul 2016 12:54:20 GMT
Alas, there are no more docs than the classes themselves, in the
lucene/replicator module, under the oal.replicator.nrt package.

Essentially, you create a PrimaryNOde (equivalent of IndexWriter) for
indexing documents, in a JVM on machine 1, and a ReplicaNode in a JVM on
machine 2, but you must subclass these classes to handle sending files
across the wire.

The test cases give simplistic examples (thread-per-socket-connection) of
how to do this.

Mike McCandless

http://blog.mikemccandless.com

On Mon, Jul 4, 2016 at 8:10 AM, Desteny Child <myshareit@gmail.com> wrote:

> Hi Mike,
>
> Thanks you very much for your response.
>
> I would be really grateful if you can please provide me with an information
> where I can read(may be with examples) about new near-real-time replication
> ?
>
> Thanks,
> Alex
>
> 2016-07-04 12:57 GMT+03:00 Michael McCandless <lucene@mikemccandless.com>:
>
> > NFS is dangerous if different nodes may take turns writing to the shared
> > index.
> >
> > Locking sometimes doesn't work correctly, client-side metadata caching
> > (e.g. the directory entry) can cause problems, NFS doesn't support
> "delete
> > on final close" semantics that Lucene relies on.
> >
> > rsync-like behavior can work with IndexWriter if you use
> > SnapshotDeletionPolicy to hold a point-in-time view of the index open for
> > copying ... this is also how to take a live backup of a still-writing
> > index, and it's how Lucene's replication module works.
> >
> > You could also try the new near-real-time replication, which copies just
> > the newly written segment files without requiring a full commit (fsync)
> on
> > the source index.
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> > On Sun, Jul 3, 2016 at 2:09 PM, Desteny Child <myshareit@gmail.com>
> wrote:
> >
> > > I need to organize a cluster for my stateless application based on
> Lucene
> > > 5.2.1. Right now I'm looking for a solution in order to share Lucene
> > index
> > > via NFS or rsync between different Lucene nodes.
> > >
> > > Is it a good idea to use NFS for this purpose and if so will it be
> > possible
> > > to read/write from different nodes to the same shared index ?
> > >
> > > Also I read that rsync tool can be used for this purpose(in order to
> > > synchronize index files across all nodes) but I can't find any success
> > > story for using rsync + Lucene. Right now I have a lot of question, one
> > of
> > > them - is it safe to use rsync at anytime especially when IndexWriter
> is
> > in
> > > progress(not closed) and actively indexes documents.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message