lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Lucene directory copy - master copy to local index
Date Mon, 22 Jun 2009 09:09:09 GMT
Or if you don't want to use Solr, rsync is very good for this.  Only takes
changes, takes care of deletions, very robust.  That is what Solr uses.

See http://wiki.apache.org/solr/CollectionDistribution for details or ideas.


--
Ian.

On Sun, Jun 21, 2009 at 4:32 AM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:

>
> Hi Amin,
>
> Have a look at Solr, it may be what you are after:
> http://lucene.apache.org/solr/
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
> > From: Amin Mohammed-Coleman <aminmc@gmail.com>
> > To: java-user@lucene.apache.org
> > Sent: Saturday, June 20, 2009 1:32:18 PM
> > Subject: Lucene directory copy - master copy to local index
> >
> > Hi
> > I am prototyping the following situation:
> >
> > 1) Multiple nodes in a cluster
> > 2) Each node has a local index
> > 3) Search requests are maded against the local index
> > 4) Index updates are sent to a JMS where a master process adds document
> to
> > index
> > 5) Each node is configured to check whether the local index is out of
> date
> > and needs to copy from master index.
> >
> > I have created a class that will be configured (quartz job) to copy
> master
> > directory files to the local directory.  I was wondering if I could get
> > advice on whether this is the best way to copy master directory changes
> to
> > the local.
> >
> > public class LocalIndexProvider {
> >
> > private Directory masterDirectory;
> >
> > private Directory localDirectory;
> >
> > private final Logger LOGGER = Logger.getLogger(getClass());
> >
> > public LocalIndexProvider(Directory masterDirectory, Directory
> > localDirectory) {
> >
> > this.masterDirectory  = masterDirectory;
> >
> > this.localDirectory = localDirectory;
> >
> > }
> >
> > public void updateLocalIndex() throws Exception {
> >
> > StopWatch stopWatch = new StopWatch("copy-time");
> >
> > stopWatch.start();
> >
> > String[] masterFiles = masterDirectory.list();
> >
> > String[] localFiles = localDirectory.list();
> >
> > Lock lock = localDirectory.getLockFactory().makeLock("test");
> >
> > lock.obtain();
> >
> > try {
> >
> > if (localFiles.length != masterFiles.length) {
> >
> > Directory.copy(masterDirectory, localDirectory, false);
> >
> > }
> >
> > }finally {
> >
> > lock.release();
> >
> > }
> >
> > stopWatch.stop();
> >
> > LOGGER.debug("total time taken :" + stopWatch.getTotalTimeMillis());
> >
> > }
> >
> > }
> >
> >
> >
> > The above is just a proof of concept and I am open to any comments about
> it.
> > It may be inefficient or it may be.  I have looked at the source code
> from
> > hibernate search and it seems as though they copy each file from the
> > directory (master to local).  I'm not sure whether this is something I
> need
> > to do or the above (with more refinement) would be enough.
> >
> > Any help would be highly appreciated.
> >
> > Cheers
> > Amin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message