hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6957) shuffle hangs after a node manager connection timeout
Date Mon, 11 Sep 2017 20:08:00 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jason Lowe updated MAPREDUCE-6957:
    Target Version/s: 2.9.0, 3.0.0-beta1, 2.7.5, 2.8.3

> shuffle hangs after a node manager connection timeout
> -----------------------------------------------------
>                 Key: MAPREDUCE-6957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6957
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Jooseong Kim
>            Assignee: Jooseong Kim
>         Attachments: MAPREDUCE-6957.001.patch
> After a connection failure from the reducer to the node manager, shuffles started to
hang with the following message:
> {code}
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 - MergeManager returned status
WAIT ...
> {code}
> There are two problems that leads to the hang.
> Problem 1.
> When a reducer has an issue connecting to the node manager, copyFromHost may call putBackKnownMapOutput
on the same task attempt multiple times.
> There are two call sites of putBackKnownMapOutput in copyFromHost since MAPREDUCE-6303:
> 1. In the finally block of copyFromHost
> 2. In the catch block of openShuffleUrl.
> When openShuffleUrl fails to connect from the catch block in copyFromHost, it returns
> By the time openShuffleUrl returns null, putBackKnownMapOutput would have been called
already for all remaining map outputs.
> However, the finally block calls putBackKnownMapOutput one more time on the map outputs.
> Problem 2. Problem 1 causes a leak in MergeManager.
> The problem occurs when multiple fetchers get the same set of map attempt outputs to
> Different fetchers reserves memory from MergeManager in Fetcher.copyMapOutput for the
same map outputs.
> When the fetch succeeds, only the first map output gets committed through ShuffleSchedulerImpl.copySucceeded
-> InMemoryMapOutput.commit, because commit() is gated by !finishedMaps[mapIndex].
> This may lead to a condition where usedMemory > memoryLimit, while commitMemory <
> This gets the MergeManager into a deadlock where a merge is never triggered while MergeManager
cannot reserve additional space for map outputs.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message