hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15887) Add an option to avoid writing data locally in Distcp
Date Thu, 01 Nov 2018 10:02:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671388#comment-16671388

Steve Loughran commented on HADOOP-15887:

one checkstyle complaint 

  public Builder withNoLocalWrite(boolean noLocalWrite) {:45: 'noLocalWrite' hides a field.

> Add an option to avoid writing data locally in Distcp
> -----------------------------------------------------
>                 Key: HADOOP-15887
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15887
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: tools/distcp
>    Affects Versions: 2.8.2, 3.0.0
>            Reporter: Tao Jie
>            Assignee: Tao Jie
>            Priority: Major
>         Attachments: HADOOP-15887.001.patch
> When copying large amount of data from one cluster to another via Distcp, and the Distcp
jobs run in the target cluster, the datanode local usage would be imbalanced. Because the
default placement policy chooses the local node to store the first replication.
> In https://issues.apache.org/jira/browse/HDFS-3702 we add a flag in DFSClient to avoid
replicating to the local datanode.  We can make use of this flag in Distcp.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message