hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13496) Include file lengths in Mismatch in length error for distcp
Date Thu, 19 Jan 2017 11:21:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829748#comment-15829748
] 

Steve Loughran commented on HADOOP-13496:
-----------------------------------------

sorry, must have applied it to branch-2 with some other applied patch in, then looked @ the
diff

> Include file lengths in Mismatch in length error for distcp
> -----------------------------------------------------------
>
>                 Key: HADOOP-13496
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13496
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: distcp
>         Attachments: HADOOP-13496.v1.patch, HADOOP-13496.v1.patch
>
>
> Currently RetriableFileCopyCommand doesn't show the perceived lengths in Mismatch in
length error:
> {code}
> 2016-08-12 10:23:14,231 ERROR [LocalJobRunner Map Task Executor #0] util.RetriableCommand(89):
Failure in Retriable command: Copying hdfs://localhost:53941/user/tyu/test-data/dc7c674a-c463-4798-8260-
  c5d1e3440a4b/WALs/10.22.9.171,53952,1471022508087/10.22.9.171%2C53952%2C1471022508087.regiongroup-1.1471022510182
to hdfs://localhost:53941/backupUT/backup_1471022580616/WALs/10.22.9.                   171%2C53952%2C1471022508087.regiongroup-1.1471022510182
> java.io.IOException: Mismatch in length of source:hdfs://localhost:53941/user/tyu/test-data/dc7c674a-c463-4798-8260-c5d1e3440a4b/WALs/10.22.9.171,53952,1471022508087/10.22.9.171%2C53952%2C1471022508087.regiongroup-1.1471022510182
and target:hdfs://localhost:53941/backupUT/backup_1471022580616/WALs/.distcp.tmp.attempt_local344329843_0006_m_000000_0
>   at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193)
>   at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126)
>   at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
>   at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
>   at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281)
>   at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
>   at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
> {code}
> It would be helpful to include what's the expected length and what's the real length.
> Thanks to [~yzhangal] for offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message