hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13002) distcp behaves differently through code compared to toolrunner invocation from command-line
Date Wed, 11 Oct 2017 23:15:03 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201157#comment-16201157

Ashutosh Chauhan commented on HADOOP-13002:

This will be useful for hive. Currently because of this bug we need to invoke distcp via tool
runner, instead of via api.
cc: [~jnp]

> distcp behaves differently through code compared to toolrunner invocation from command-line
> -------------------------------------------------------------------------------------------
>                 Key: HADOOP-13002
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13002
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tools/distcp
>    Affects Versions: 2.5.0, 2.6.0, 2.7.0, 3.0.0-alpha1
>            Reporter: Joep Rottinghuis
>            Assignee: Gary Steelman
> In Hadoop 2.5 the behavior of distcp changed when called through code iff the target
directory did not exist and update wasn't used and atomic wasn't used.
> HADOOP-10459 introduced a change to preserve the root directory attributes. It introduced
a derivative property in the options as well as in the configuration whether the target path
exists. See https://github.com/apache/hadoop/commit/c5b59477775c797944db4992e8a70289ba2895ed
> However, this property is set only when distcp is used through the command line as a
ToolRunner in Distcp.run(String[] argv).
> The result is that when the target directory doesn't exist (and neither -update nor -atomic
options are used) SimplyCopyListing incorrectly assumes that the target directory does exist
because the attribute defaults to true. Copying directory a/b/c to xyz results in the creation
of a xyx/c directory with the content of c in it, rather than the content of c getting copied
into directory xyz directly.  

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message