incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rahul Challapalli (JIRA)" <>
Subject [jira] [Commented] (BLUR-132) Create Index Snapshots
Date Tue, 13 Aug 2013 05:28:48 GMT


Rahul Challapalli commented on BLUR-132:

We can provide the backup in multiple ways :
     1. Backup all shards(all tables) on a shard server onto local filesystem(cluster/table/shard/files).
While restoring from backup, every shard server
        reads from its local filesystem and copies the shards onto HDFS.
     2. Bakup all shards on all shard servers onto a common HDFS location. While restoring
we would partition the shards onto shard servers.
We should also have some mechanism to restore from a backup. For us to restore the index from
a backup, we might as well need a
point-in-time copy of all the table descriptors.

How are we planning to expose this snapshot functionality (Shell, API, BOTH)?

Where are we even using LocalIndexServer?

I was able to take a backup by wrapping IndexDeletionPolicy with SnapshotDeletionPolicy and
then take a snapshot
and copy all the files to a local file system. This technically works even if the index is
being actively updated, but the way in
which the code is structured (DistributeIndexServer.openShard), we would only get a BlurIndexReader
when the shard is being updated.
but the sample code I have below is using the writer to take the snapshot. May be there is
a different way?

                           * As of now we cannot use the snapshotDeletionPolicy instance which
we sent to IndexWriter 
			   * as it is getting cloned internally by IndexWriter. So we always get it from the LiveIndexWriterConfig
			  SnapshotDeletionPolicy snapshotter = (SnapshotDeletionPolicy)writer.getConfig().getIndexDeletionPolicy();
			  String commitIdentifier = generateCommitIdentifier();
			  IndexCommit indexCommit = snapshotter.snapshot(commitIdentifier);
			  for (String filename : indexCommit.getFileNames()) {
			    if (!filename.equals("write.lock")) {
			      // take backup
Also What happens when multiple sources try to add documents to the same shard simultaneously(using
the same IndexWriter)?

Would really love to know your thoughts and appreciate it if someone can fill in gaps in my
understanding. Thank You.
> Create Index Snapshots
> ----------------------
>                 Key: BLUR-132
>                 URL:
>             Project: Apache Blur
>          Issue Type: New Feature
>    Affects Versions: 0.3.0
>            Reporter: Aaron McCurry
>             Fix For: 0.3.0

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message