hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9820) Improve distcp to support efficient restore to an earlier snapshot
Date Thu, 06 Oct 2016 15:35:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552241#comment-15552241

Yongjun Zhang commented on HDFS-9820:

Hi [~andrew.wang],

As I verbally addressed your question
Is there a situation where we'd want to pass two different clusters for "src" and "tgt" to
rdiff? They need to both have identical "s1" base states anyway.
The reason is that a user reported copying from a different cluster is faster,  so I hope
we can support the flexibility to copy from either the same cluster's snapshot or a different
cluster's snapshot. It's common for user to have a mirroring cluster, one is production, one
is a backup.

I will address most of the checkstyle issue in next rev, but one thing I followed the existing
test is to use "_" in file name variables to indicate the hierarchy, which seems easier to
read, so I'd like to keep that.


> Improve distcp to support efficient restore to an earlier snapshot
> ------------------------------------------------------------------
>                 Key: HDFS-9820
>                 URL: https://issues.apache.org/jira/browse/HDFS-9820
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: distcp
>    Affects Versions: 2.6.4
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>         Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch, HDFS-9820.003.patch, HDFS-9820.004.patch,
HDFS-9820.005.patch, HDFS-9820.006.patch
> A common use scenario (scenaio 1): 
> # create snapshot sx in clusterX, 
> # do some experiemnts in clusterX, which creates some files. 
> # throw away the files changed and go back to sx.
> Another scenario (scenario 2) is, there is a production cluster and a backup cluster,
we periodically sync up the data from production cluster to the backup cluster with distcp.

> The cluster in scenario 1 could be the backup cluster in scenario 2.
> For scenario 1:
> HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are some complexity
and challenges.  Before that jira is implemented, we count on distcp to copy from snapshot
to the current state. However, the performance of this operation could be very bad because
we have to go through all files even if we only changed a few files.
> For scenario 2:
> HDFS-7535 improved distcp performance by avoiding copying files that changed name since
last backup.
> On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data from source
to target cluster, by only copying changed files since last backup. The way it works is use
snapshot diff to find out all files changed, and copy the changed files only.
> See https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/
> This jira is to propose a variation of HDFS-8828, to find out the files changed in target
cluster since last snapshot sx, and copy these from snapshot sx of either the source or the
target cluster, to restore target cluster's current state to sx. 
> Specifically,
> If a file/dir is
> - renamed, rename it back
> - created in target cluster, delete it
> - modified, put it to the copy list
> - run distcp with the copy list, copy from the source cluster's corresponding snapshot
> This could be a new command line switch -rdiff in distcp.
> As a native restore feature, HDFS-4167 would still be ideal to have. However,  HDFS-9820
would hopefully be easier to implement, before HDFS-4167 is in place.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message