hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13593) `hadoop distcp -atomic` invokes improper host check while copying data from HDFS to Swift
Date Mon, 12 Sep 2016 09:17:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483594#comment-15483594

Steve Loughran commented on HADOOP-13593:

# please can you stick the full stack trace of the exception in as a comment; good for future
searches of the same lines, & IDEs that navigate up the stack. Leaving out the bug description
—as you did— is great, because it keeps out of all the emails, but having that stack
in the comments is always useful
# anything checking hostnames is going to be there for a reason. Presumably so that people
can use URLs like hdfs://server:4040/ and hdfs://server.example.org:4040/ and have atomic
operations. Changing this fairly fundamental behaviour is not going to happen, because it
affects so much more than swift For that reason, it's going to have to be a -1 there, sorry.
# none of the object stores support atomic renames, so an atomic distcp isn't going to work.
in fact, maybe they should all reject the option outright to stop people thinking of it.

If there were to be a patch on this, it'd need tests. Here I'd recommend that (a) swift adds
an implementation of {{AbstractContractDistCpTest}}, and that (b), that base test added a
check for the atomic flag —one that every fs contract XML would have to declare whether
or not it supported. The option. 

> `hadoop distcp -atomic` invokes improper host check while copying data from HDFS to Swift
> -----------------------------------------------------------------------------------------
>                 Key: HADOOP-13593
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13593
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Yuanbo Liu
>         Attachments: HADOOP-13593.001.patch, HADOOP-13593.002.patch
> While copying data from HDFS to Swift by using `hadoop distcp -atomic`, for example:
> {code}
> hadoop distcp -atomic /tmp/100M  swift://testhadoop.softlayer//tmp
> {code}
> it throws
> {code}
> java.lang.IllegalArgumentException: Work path swift://testhadoop.softlayer/._WIP_tmp546958075
and target path swift://testhadoop.softlayer/tmp are in different file system
> 	at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:351)
> .....
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message