hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (MAPREDUCE-7282) MR v2 commit algorithm should be deprecated and not the default
Date Thu, 10 Jun 2021 20:59:01 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-7282?focusedWorklogId=609929&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609929

ASF GitHub Bot logged work on MAPREDUCE-7282:

                Author: ASF GitHub Bot
            Created on: 10/Jun/21 20:58
            Start Date: 10/Jun/21 20:58
    Worklog Time Spent: 10m 
      Work Description: steveloughran commented on pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#issuecomment-859061589

   Once the manifest committer #2971 is in, the abfs and gcs stores will get something faster
than v1 but with its task failure semantics
   * all treewalking for task attempt listing will be in task commit
   * no dir renames in task commit, just saving of the manifest
   * job commit: parallel load of manifests, merge of list of directories to create, parallel
set of mkdirs and then the parallelized renames. 
   In this world, we can just leave people using v1/v2 alone, and for (spark) jobs in azure
and google cloud say "use the manifest committer". 
   Which means I can just close this as a wontfix.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:

Issue Time Tracking

            Worklog Id:     (was: 609929)
    Remaining Estimate: 0h
            Time Spent: 10m

> MR v2 commit algorithm should be deprecated and not the default
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-7282
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7282
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 3.3.0, 3.2.1, 3.1.3, 3.3.1
>            Reporter: Steve Loughran
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
> The v2 MR commit algorithm moves files from the task attempt dir into the dest dir on
task commit -one by one
> It is therefore not atomic
> # if a task commit fails partway through and another task attempt commits -unless exactly
the same filenames are used, output of the first attempt may be included in the final result
> # if a worker partitions partway through task commit, and then continues after another
attempt has committed, it may partially overwrite the output -even when the filenames are
the same
> Both MR and spark assume that task commits are atomic. Either they need to consider that
this is not the case, we add a way to probe for a committer supporting atomic task commit,
and the engines both add handling for task commit failures (probably fail job)
> Better: we remove this as the default, maybe also warn when it is being used

This message was sent by Atlassian Jira

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message