hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-7282) MR v2 commit algorithm should be deprecated and not the default
Date Thu, 10 Jun 2021 20:59:01 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

ASF GitHub Bot updated MAPREDUCE-7282:
    Labels: pull-request-available  (was: )

> MR v2 commit algorithm should be deprecated and not the default
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-7282
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7282
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 3.3.0, 3.2.1, 3.1.3, 3.3.1
>            Reporter: Steve Loughran
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
> The v2 MR commit algorithm moves files from the task attempt dir into the dest dir on
task commit -one by one
> It is therefore not atomic
> # if a task commit fails partway through and another task attempt commits -unless exactly
the same filenames are used, output of the first attempt may be included in the final result
> # if a worker partitions partway through task commit, and then continues after another
attempt has committed, it may partially overwrite the output -even when the filenames are
the same
> Both MR and spark assume that task commits are atomic. Either they need to consider that
this is not the case, we add a way to probe for a committer supporting atomic task commit,
and the engines both add handling for task commit failures (probably fail job)
> Better: we remove this as the default, maybe also warn when it is being used

This message was sent by Atlassian Jira

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message