hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15469) S3A directory committer commit job fails if _temporary directory created under dest
Date Tue, 15 May 2018 15:55:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476055#comment-16476055
] 

Steve Loughran commented on HADOOP-15469:
-----------------------------------------

Patch 001. Remove test, fix mock tests to verify that the behaviour has now changed. There's
still an existence check in job setup, we just don't overreact otherwise. The alternative
would be a more complex & brittle scan for >1 non-temp entry. I think that's overkill

Testing: s3 ireland. First run failed with HADOOP-14946 over a slow network & many workers;
rerun made it go away.

FYI, [~rdblue]

> S3A directory committer commit job fails if _temporary directory created under dest
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-15469
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15469
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>         Environment: spark test runs
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-15469-001.patch
>
>
> The directory staging committer fails in commit job if any temporary files/dirs have
been created. Spark work can create such a dir for placement of absolute files.
> This is because commitJob() looks for the dest dir existing, not containing non-hidden
files.
> As the comment says, "its kind of superfluous". More specifically, it means jobs which
would commit with the classic committer & overwrite=false will fail
> Proposed fix: remove the check



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message