hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented
Date Thu, 13 Oct 2016 18:17:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572757#comment-15572757

Jing Zhao commented on HADOOP-13024:

.8 patch looks pretty good to me. The test code refactoring looks great. Only one nitpicky
# All the constant paths can be defined in DistCpConstants.java as following:
  public static final String NONE_PATH_NAME = "/NONE";
  public static final Path NONE_PATH = new Path(NONE_PATH_NAME);
  public static final Path RAW_NONE_PATH = new Path(

+1 after addressing the comment.

> Distcp with -delete feature on raw data not implemented
> -------------------------------------------------------
>                 Key: HADOOP-13024
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13024
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Mavin Martin
>            Assignee: Mavin Martin
>         Attachments: HADOOP-13024.patch, HADOOP-13024.patch, HADOOP-13024.patch.3, HADOOP-13024.patch.4,
HADOOP-13024.patch.5, HADOOP-13024.patch.6, HADOOP-13024.patch.7, HADOOP-13024.patch.8
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit failed: org.apache.hadoop.tools.CopyListing$InvalidInputException:
The source path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the target
path 'hdfs://nn/NONE' does not. Either all or none of the paths must have this prefix.
>         at org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
>         at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
>         at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
>         at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
>         at org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
>         at org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
>         at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
>         at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>         at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
>         at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to delete things
in the target that no longer exist in the source, it revalidates to make sure NONE is in the
/.reserved/raw domain.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message