hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6512) FileOutputCommitter tasks unconditionally create parent directories
Date Mon, 11 Apr 2016 21:36:25 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated MAPREDUCE-6512:
----------------------------------
    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

bq. should we close this as won't fix?

Agree that this is probably too risky to fix given all the custom output committers that could
be implicitly relying on this behavior.

> FileOutputCommitter tasks unconditionally create parent directories
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6512
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6512
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Chang Li
>            Assignee: Chang Li
>         Attachments: MAPREDUCE-6512.2.patch, MAPREDUCE-6512.2.patch, MAPREDUCE-6512.patch,
MAPREDUCE-6512.patch
>
>
> If the output directory is deleted then subsequent tasks should fail. Instead they blindly
create the missing parent directories, leading the job to be "succesful" despite potentially
missing almost all of the output. Task attempts should fail if the parent app attempt directory
is missing when they go to create their task attempt directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message