incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rahul Challapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BLUR-132) Create Index Snapshots
Date Wed, 14 Aug 2013 07:12:53 GMT

    [ https://issues.apache.org/jira/browse/BLUR-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739336#comment-13739336
] 

Rahul Challapalli commented on BLUR-132:
----------------------------------------

Thanks for the reply Aaron. I understood the task differently(to take backups). Thanks for
the clarification. Below are few of my thoughts.

We might not need to add/modify any IndexDeletionPolicy if my understanding is correct

We can use the SnapshotDeletionPolicy(wrapping KeepOnlyLastCommittedIndexDeletionPolicy) to
snapshot the last commit. 
SnapshotDeletionPolicy controls what IndexCommit's are passed to KeepOnlyLastCommittedIndexDeletionPolicy
and does not include the
snapshotted IndexCommits. If we do not call the release method then the snapshotted index
commit will not be deleted

This is how we instantiate the SnapshotDeletionPolicy (so that snapshotted IndexCommits do
not get deleted upon shardserver restart)
SnapshotDeletionPolicy snapshotter = new SnapshotDeletionPolicy(new KeepOnlyLastCommittedIndexDeletionPolicy(),
snapshotsInfo);

Below is some pseudo code

DistributedIndexServer {
  Map<String, String> snapshotsInfo; // snapshotId --> segmentsFilename
  
  public void init() {
    ...
    ...
    loadSnapshotsInfo(); 
  }
  
  public String snapshot() {
    try {
      for(String table : _clusterStatus.getTableList(true)) {
        Map<String, BlurIndex> indexes = getIndexes(table);
        for (String shard : indexes.keySet()) {
          BlurIndex index = indexes.get(shard);
          // Not sure how to get the writer.....BlurIndexReader can return null may be?
          BlurIndexWriter writer = index.getWriter();
          if (writer != null) {
            writer.commit();
            SnapshotDeletionPolicy snapshotter = (SnapshotDeletionPolicy)writer.getConfig().getIndexDeletionPolicy();
            String commitIdentifier = generateCommitIdentifier();
            IndexCommit indexCommit = snapshotter.snapshot(commitIdentifier);
            _snapshotsInfo.put(commitIdentifier, indexCommit.getSegmentsFileName());
            // not sure where to persistently store _snapshotsInfo to load it on shard server
restart
            return commitIdentifier;
          } else {
            // index is being updated....we have an instance of BlurIndexReader
            LOG.error("Cannot take a snapshot as the Index is being updated on " + table +
"/" + shard);
            throw new IOException("Cannot take a snapshot as the Index is being updated on
" + table + "/" + shard);
          }
        }
      }
    } catch (Exception e) {
      LOG.error("Unknown error while trying to get a table list.", e);
      throw new IOException("Unknown error while trying to get a table list.", e);
    }
  }
  
  private String generateCommitIdentifier() {
	  return UUID.randomUUID().toString();
  }
}

If we want to read from a particular snapshot we can use DirectoryReader.open(indexCommit)
in BlurIndexReader. 

Sorry for the huge comment and random pseudo code. I don't want to get too far if I am missing
something

- Rahul
  

                
> Create Index Snapshots
> ----------------------
>
>                 Key: BLUR-132
>                 URL: https://issues.apache.org/jira/browse/BLUR-132
>             Project: Apache Blur
>          Issue Type: New Feature
>    Affects Versions: 0.3.0
>            Reporter: Aaron McCurry
>             Fix For: 0.3.0
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message