lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Question on Lucene hot-backup functionality.
Date Wed, 24 Jul 2013 11:33:49 GMT
This is unfortunately very trappy ... this happened with LUCENE-4876,
where we added cloning of IndexDeletionPolicy on IW construction.
It's very confusing that the IDP you set on your IWC is not in fact
the one that IW uses...

Mike McCandless

http://blog.mikemccandless.com


On Wed, Jul 24, 2013 at 2:35 AM, Shai Erera <serera@gmail.com> wrote:
> Hi
>
> In Lucene 4.4 we've improved the snapshotting process so that you don't
> need to specify an ID.
> Also, there's a new Replicator module which can be used for just that
> purpose - take hot index backups of the index.
> It pretty much hides most of the snapshotting from you. You can read about
> it here: http://shaierera.blogspot.com/2013/05/the-replicator.html
>
> As for your problem, I think it's related to the fact IndexWriter clones
> the given IndexWriterConfig, including the SnapshotDeletionPolicy.
> So you should obtain it from IW.getLiveConfig().getIndexDeletionPolicy(),
> rather than IndexWriterConfig.getIndexDeletionPolicy(). I'm not sure what
> Indexer.getSnapshotter() does, but I'd make sure that it uses IW.
>
> Shai
>
>
> On Wed, Jul 24, 2013 at 7:34 AM, Marcos Juarez Lopez <mjuarez@gmail.com>wrote:
>
>> I'm trying to get Lucene's hot backup functionality to work.  I posted the
>> question in detail over at StackOverflow, but it seems there's very little
>> Lucene knowledge over there.
>>
>> Basically, I think I have setup everything correctly, but I can't get a
>> valid snapshot when trying to do a backup.  I'm following both the Lucene
>> book's instructions, as well as the latest Lucene Javadocs, to no avail.
>>  Original question at the link, but I'll copy the relevant bits below:
>>
>> http://stackoverflow.com/questions/17753226/lucene-4-3-1-backup-process
>>
>> This is the code I have up to now:
>>
>> public Indexer(Directory indexDir, PrintStream printStream) throws
>> IOException {
>>     IndexWriterConfig config = new
>> IndexWriterConfig(Version.LUCENE_43, new Analyzer());
>>     snapshotter = new SnapshotDeletionPolicy(new
>> KeepOnlyLastCommitDeletionPolicy());
>>     writerConfig.setIndexDeletionPolicy(snapshotter);
>>     indexWriter = new IndexWriter(indexDir, writerConfig);
>> }
>>
>> And when starting the backup, you can't just do snapshotter.snapshot(). You
>> now have to specify an arbitrary commitIdentifier id, and use that after
>> you're done to release the snapshot.
>>
>> SnapshotDeletionPolicy snapshotter = indexer.getSnapshotter();
>> String commitIdentifier = generateCommitIdentifier();
>> try {
>>     IndexCommit commit = snapshotter.snapshot(commitIdentifier);
>>     for (String fileName : commit.getFileNames()) {
>>         backupFile(fileName);
>>     }
>> } catch (Exception e) {
>>     logger.error("Exception", e);
>> } finally {
>>     snapshotter.release(commitIdentifier);
>>     indexer.deleteUnusedFiles();
>> }
>>
>> However, this doesn't seem to be working. Regardless of whether there have
>> been docs indexed or not, and regardless of whether I have committed or
>> not, my call tosnapshotter.snapshot(commitIdentifier) always throws an
>> IllegalStateException sayingNo index commit to snapshot. Looking at the
>> code, the SnapshotDeletionPolicy seems to think there have been no commits,
>> even though I'm committing to disk every 5 seconds or so. I've verified,
>> and there are docs being written and committed to indexes all the time, but
>> snapshotter always thinks there have been zero commits.
>>
>> Any idea of what I'm doing wrong?
>>
>> Thanks!
>>
>> Marcos Juarez
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message