hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
Date Wed, 08 Nov 2017 13:23:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243916#comment-16243916

Steve Loughran commented on HADOOP-15003:

Side issue, one for [~rdblue] in particular

If the job commit fails, should any existing {{_SUCCESS}} file be deleted? Because things
may be inconsistent & we shouldn't have any cue in the dir that it contains valid data.

What I'd do is: delete the marker before any files are committed or deleted, but after any
conflict resolution checks which may fail the job have run. That way, if a job commit is aborted:
the existing marker is unchanged.
* directory committer: delete it for the APPEND case; retain for FAIL (its deleted for free
* partitioned committer: both OVERWRITE and APPEND to delete the marker, though append will

Moot for the magic commit as it doesn't do any in-situ overwrites/appends of data.

If you think we should do this, I can add it as a folowup JIRA; would need test modification

> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>                 Key: HADOOP-15003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15003
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, HADOOP-13786-043.patch,
HADOOP-13786-044.patch, HADOOP-13786-045.patch, HADOOP-13786-046.patch
> This is a Yetus only JIRA created to have Yetus review the HADOOP-13786/HADOOP-14971
patch as a .patch file, as the review PR [https://github.com/apache/hadoop/pull/282] is stopping
this happening in HADOOP-14971.
> Reviews should go into the PR/other task

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message