lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcos Juarez Lopez <mjua...@gmail.com>
Subject Re: Question on Lucene hot-backup functionality.
Date Thu, 25 Jul 2013 19:15:48 GMT
Shai and Michael,

Thanks for your help.  That did the trick, getting the live
IndexDeletionPolicy from the live writer config.

I also agree with Michael, it's not obvious at all from the docs and/or
Lucene in Action book.  Maybe this can be clarified on the
IndexDeletionPolicy javadocs going forward?

Thanks,

Marcos Juarez



On Wed, Jul 24, 2013 at 5:33 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> This is unfortunately very trappy ... this happened with LUCENE-4876,
> where we added cloning of IndexDeletionPolicy on IW construction.
> It's very confusing that the IDP you set on your IWC is not in fact
> the one that IW uses...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, Jul 24, 2013 at 2:35 AM, Shai Erera <serera@gmail.com> wrote:
> > Hi
> >
> > In Lucene 4.4 we've improved the snapshotting process so that you don't
> > need to specify an ID.
> > Also, there's a new Replicator module which can be used for just that
> > purpose - take hot index backups of the index.
> > It pretty much hides most of the snapshotting from you. You can read
> about
> > it here: http://shaierera.blogspot.com/2013/05/the-replicator.html
> >
> > As for your problem, I think it's related to the fact IndexWriter clones
> > the given IndexWriterConfig, including the SnapshotDeletionPolicy.
> > So you should obtain it from IW.getLiveConfig().getIndexDeletionPolicy(),
> > rather than IndexWriterConfig.getIndexDeletionPolicy(). I'm not sure what
> > Indexer.getSnapshotter() does, but I'd make sure that it uses IW.
> >
> > Shai
> >
> >
> > On Wed, Jul 24, 2013 at 7:34 AM, Marcos Juarez Lopez <mjuarez@gmail.com
> >wrote:
> >
> >> I'm trying to get Lucene's hot backup functionality to work.  I posted
> the
> >> question in detail over at StackOverflow, but it seems there's very
> little
> >> Lucene knowledge over there.
> >>
> >> Basically, I think I have setup everything correctly, but I can't get a
> >> valid snapshot when trying to do a backup.  I'm following both the
> Lucene
> >> book's instructions, as well as the latest Lucene Javadocs, to no avail.
> >>  Original question at the link, but I'll copy the relevant bits below:
> >>
> >> http://stackoverflow.com/questions/17753226/lucene-4-3-1-backup-process
> >>
> >> This is the code I have up to now:
> >>
> >> public Indexer(Directory indexDir, PrintStream printStream) throws
> >> IOException {
> >>     IndexWriterConfig config = new
> >> IndexWriterConfig(Version.LUCENE_43, new Analyzer());
> >>     snapshotter = new SnapshotDeletionPolicy(new
> >> KeepOnlyLastCommitDeletionPolicy());
> >>     writerConfig.setIndexDeletionPolicy(snapshotter);
> >>     indexWriter = new IndexWriter(indexDir, writerConfig);
> >> }
> >>
> >> And when starting the backup, you can't just do snapshotter.snapshot().
> You
> >> now have to specify an arbitrary commitIdentifier id, and use that after
> >> you're done to release the snapshot.
> >>
> >> SnapshotDeletionPolicy snapshotter = indexer.getSnapshotter();
> >> String commitIdentifier = generateCommitIdentifier();
> >> try {
> >>     IndexCommit commit = snapshotter.snapshot(commitIdentifier);
> >>     for (String fileName : commit.getFileNames()) {
> >>         backupFile(fileName);
> >>     }
> >> } catch (Exception e) {
> >>     logger.error("Exception", e);
> >> } finally {
> >>     snapshotter.release(commitIdentifier);
> >>     indexer.deleteUnusedFiles();
> >> }
> >>
> >> However, this doesn't seem to be working. Regardless of whether there
> have
> >> been docs indexed or not, and regardless of whether I have committed or
> >> not, my call tosnapshotter.snapshot(commitIdentifier) always throws an
> >> IllegalStateException sayingNo index commit to snapshot. Looking at the
> >> code, the SnapshotDeletionPolicy seems to think there have been no
> commits,
> >> even though I'm committing to disk every 5 seconds or so. I've verified,
> >> and there are docs being written and committed to indexes all the time,
> but
> >> snapshotter always thinks there have been zero commits.
> >>
> >> Any idea of what I'm doing wrong?
> >>
> >> Thanks!
> >>
> >> Marcos Juarez
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message