hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12469) distcp Ignore failures option ignored
Date Fri, 09 Oct 2015 22:07:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14951286#comment-14951286

Mingliang Liu commented on HADOOP-12469:

Hi [~jira.shegalov], thanks for reporting this.

I'm working on a patch to wrap the {{CopyReadException}} only once. However, I found it's
hard to write a unit test which will fail. The current unit tests are not able to detect this
bug because the {{map}} method throws an {{CopyReadException}} when calling {{getFileStatus}},
before it calls the {{copyFileWithRetry}}. This makes sense as the file is deleted before
the copy operation. This way the unit tests pass with the following condition true (in case
of ignoreFailures):
if (ignoreFailures && exception.getCause() instanceof

Should we address the orthogonal {{ignoreFailures}} mutually exclusive with the atomic option

> distcp Ignore failures option ignored
> -------------------------------------
>                 Key: HADOOP-12469
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12469
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tools/distcp
>    Affects Versions: 2.7.1
>            Reporter: Gera Shegalov
>            Assignee: Mingliang Liu
>            Priority: Critical
> {{RetriableFileCopyCommand.CopyReadException}} is double-wrapped via
> # via {{RetriableCommand::execute}}
> # via {{CopyMapper#copyFileWithRetry}}
> before {{CopyMapper::handleFailure}} tests 
> {code}
> if (ignoreFailures && exception.getCause() instanceof
>             RetriableFileCopyCommand.CopyReadException
> {code}
> which is always false.
> Orthogonally, ignoring failures should be mutually exclusive with the atomic option otherwise
an incomplete dir is eligible for commit defeating the purpose.

This message was sent by Atlassian JIRA

View raw message