hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9820) Improve distcp to support efficient restore to an earlier snapshot
Date Mon, 28 Mar 2016 18:55:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214680#comment-15214680

Jing Zhao commented on HDFS-9820:

Thanks for working on this, [~yzhangal]! Some comments on the current patch:
# For the rdiff, We can switch the order of from-snapshot and to-snapshot before computing
the diff report on the target cluster. In this way we can reuse the original sync code.
# Currently rdiff is a standalone option for distcp. This means we're using distcp to do the
restore. To restore a directory back to a snapshot, this may not be the most efficient way
compared with a local restoring solution (HDFS-4167), which can avoid most of the unnecessary
data copying and can provide a copy-on-write semantic when supporting restoring appended/truncated
# But before we finish the work in HDFS-4167, maybe we can augment the current diff-based
distcp by allowing the admin to choose to restore the target back to the latest snapshot.
We can still use the implementation in the current patch, but instead of adding a new rdiff
option for distcp, we add a "--force" option to the current diff-based distcp. What do you

> Improve distcp to support efficient restore to an earlier snapshot
> ------------------------------------------------------------------
>                 Key: HDFS-9820
>                 URL: https://issues.apache.org/jira/browse/HDFS-9820
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: distcp
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>         Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch
> HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are some complexity
and challenges. 
> HDFS-7535 improved distcp performance by avoiding copying files that changed name since
last backup.
> On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data from source
to target cluster, by only copying changed files since last backup. The way it works is use
snapshot diff to find out all files changed, and copy the changed files only.
> See https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/
> This jira is to propose a variation of HDFS-8828, to find out the files changed in target
cluster since last snapshot sx, and copy these from the source target's same snapshot sx,
to restore target cluster to sx.
> If a file/dir is
> - renamed, rename it back
> - created in target cluster, delete it
> - modified, put it to the copy list
> - run distcp with the copy list, copy from the source cluster's corresponding snapshot
> This could be a new command line switch -rdiff in distcp.
> HDFS-4167 would still be nice to have. It just seems to me that HDFS-9820 would hopefully
be easier to implement.

This message was sent by Atlassian JIRA

View raw message