hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Jie (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-15887) Add an option to avoid writing data locally in Distcp
Date Tue, 30 Oct 2018 05:56:00 GMT
Tao Jie created HADOOP-15887:

             Summary: Add an option to avoid writing data locally in Distcp
                 Key: HADOOP-15887
                 URL: https://issues.apache.org/jira/browse/HADOOP-15887
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 3.0.0, 2.8.2
            Reporter: Tao Jie
            Assignee: Tao Jie

When copying large amount of data from one cluster to another via Distcp, and the Distcp jobs
run in the target cluster, the datanode local usage would be imbalanced. Because the default
placement policy chooses the local node to store the first replication.

In https://issues.apache.org/jira/browse/HDFS-3702 we add a flag in DFSClient to avoid replicating
to the local datanode.  We can make use of this flag in Distcp.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message