lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh parab <rajesh_para...@yahoo.com>
Subject Re: Lucene 2.3.0 and NFS
Date Thu, 10 Apr 2008 01:10:19 GMT
Hi All,

Has anyone used rsync or similar utilities on Windows
OS to replicate Lucene index across multiple machines?

Any pointers on it will be very useful?

Regards,
Rajesh

--- Rajesh parab <rajesh_parab_1@yahoo.com> wrote:

> Hi Michael,
> 
> Thanks a lot for your suggestions.
> 
> I was looking at rsync; as per this link
> (http://samba.anu.edu.au/rsync/features.html), rsync
> is a file transfer program for UNIX. Is there rsync
> support for Windows as well? I found few rsync
> programs that works for Windows, but I am not sure
> if
> they will server the purpose. Has anyone using rsync
> on Windows?
> 
> Regards,
> Rajesh
> 
> --- Michael McCandless <lucene@mikemccandless.com>
> wrote:
> 
> > 
> > Rajesh parab wrote:
> > > Hi,
> > >
> > > We are currently using Lucene 2.0 for full-text
> > > searches within our enterprise application,
> which
> > can
> > > be deployed in clustered environment. We
> generate
> > > Lucene index for data stored inside relational
> > > database.
> > >
> > > As Lucene 2.0 did not have solid NFS support and
> > as we
> > > wanted Lucene based searches to work properly in
> > > Clustered environment, we had decided on
> following
> > > approach:
> > > 1. The index generation happens on a machine
> > (could be
> > > one of the cluster nodes or a separate machine)
> > and
> > > once the Lucene index is generated, we copy all
> > the
> > > index files to the database.
> > 
> > Note that you can do also incremental replication:
> > often, the Lucene  
> > index changes in minor ways (eg a single new
> segment
> > is flushed and a  
> > new segments_N and segments.gen is written) so you
> > should only sync  
> > the files that are new (and remove the ones that
> are
> > now gone).   
> > Lucene's write-once approach makes this very
> simple
> > (you just have to  
> > compare file names, not the contents of each
> file).
> > 
> > It's also possible to replicate without using a
> DB. 
> > EG rsync does a  
> > great job.
> > 
> > > 2. The index search request on each cluster node
> > > retrieves the index files from database (during
> > first
> > > search or after index update), copies to the
> file
> > > system and use it for searches.
> > > 3. Thus, each cluster node has its own copy of
> the
> > > index and it keeps on picking up latest version
> if
> > it
> > > is available inside database.
> > >
> > > This has worked fine for us till now, though we
> > will
> > > not be able continue with this model in future
> as
> > we
> > > want to support Lucene based searches across our
> > > application and also want to index large
> > components
> > > inside our application like Wiki, forums, etc.
> As
> > the
> > > index will grow, storing and retrieving index
> > files
> > > from database will not be an efficient
> operation.
> > >
> > > My questions are:
> > > - Will we be able to use NFS if we move to
> Lucene
> > > 2.3.0?
> > 
> > Make sure you update to 2.3.1, not 2.3.0.
> > 
> > > - Will there be any significant performance
> impact
> > on
> > > index generation and searches if we move to NFS?
> > > - Is Lucene + NFS combination supported for all
> > > operating systems? (We support Windows, Solaris,
> > AIX,
> > > HP-UX, Red Hat Linux)
> > 
> > NFS *should* work, however:
> > 
> >    * It's not widely used, so, test thoroughly in
> > your particular setup.
> > 
> >    * Most likely to work is if you use a single
> > machine writing to  
> > the index, and many readers.
> > 
> >    * Performance is likely not great, especially
> on
> > searching, but  
> > you should test in your specific situation.
> > 
> > > - Is there any other alternative available other
> > than
> > > NFS?
> > >
> > 
> > It's also possible to replicate without using a
> DB. 
> > EG rsync does  
> > agreat job.
> > 
> > You should look at Solr, since it already has all
> > the infrastructure  
> > toaccept updates, replicate index changes to
> remote
> > machines, etc.
> > 
> > > I will really appreciate your comments/thoughts
> on
> > > this topic.
> > >
> > > Regards,
> > > Rajesh
> > >
> > >
> > >        
> > >
> >
>
______________________________________________________________________
> > 
> > > ______________
> > > You rock. That's why Blockbuster's offering you
> > one month of  
> > > Blockbuster Total Access, No Cost.
> > >
> http://tc.deals.yahoo.com/tc/blockbuster/text5.com
> > >
> > >
> >
>
---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > >
> > 
> > 
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > 
> > 
> 
> 
> 
>      
>
____________________________________________________________________________________
> You rock. That's why Blockbuster's offering you one
> month of Blockbuster Total Access, No Cost.  
> http://tc.deals.yahoo.com/tc/blockbuster/text5.com
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail:
> java-user-help@lucene.apache.org
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message