hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-15469) S3A directory committer commit job fails if _temporary directory created under set
Date Tue, 15 May 2018 14:45:00 GMT
Steve Loughran created HADOOP-15469:
---------------------------------------

             Summary: S3A directory committer commit job fails if _temporary directory created
under set
                 Key: HADOOP-15469
                 URL: https://issues.apache.org/jira/browse/HADOOP-15469
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.1.0
         Environment: spark test runs
            Reporter: Steve Loughran
            Assignee: Steve Loughran


The directory staging committer fails in commit job if any temporary files/dirs have been
created. Spark work can create such a dir for placement of absolute files.

This is because commitJob() looks for the dest dir existing, not containing non-hidden files.
As the comment says, "its kind of superfluous". More specifically, it means jobs which would
commit with the classic committer & overwrite=false will fail

Proposed fix: remove the check



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


Mime
View raw message