hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
Date Wed, 14 Mar 2018 15:35:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398761#comment-16398761
] 

Steve Loughran commented on HADOOP-15107:
-----------------------------------------

Patch 002. No changes in functionality other than logging and some error reporting. The things
which turn out useful in troubleshooting. No other changes to the code from patch 001

# {{StagingCommitter.failDestinationExists(path, operation)}} generates the E_DEST_EXISTS
exception when the destination dir exists. Before doing that, it lists the first 10 descendants
of the dest path. This helps you work out what's going on when a conflict check fails (wrong
output format used, something created a file, etc, etc)
# better logging of the toString aspects of the committers, and PathOutputCommitter's logging
of lifecycle events.


> Prove the correctness of the new committers, or fix where they are not correct
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-15107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15107
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch
>
>
> I'm writing about the paper on the committers, one which, being a proper paper, requires
me to show the committers work.
> # define the requirements of a "Correct" committed job (this applies to the FileOutputCommitter
too)
> # show that the Staging committer meets these requirements (most of this is implicit
in that it uses the V1 FileOutputCommitter to marshall .pendingset lists from committed tasks
to the final destination, where they are read and committed.
> # Show the magic committer also works.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message