hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6514) Job hangs as ask is not updated after ramping down of all reducers
Date Fri, 06 May 2016 00:09:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273364#comment-15273364
] 

Wangda Tan commented on MAPREDUCE-6514:
---------------------------------------

[~vinodkv],

This behavior is commented by [~rohithsharma] above in above comment,
bq. As a result, number of containers count in the ask is increased as explained in the below...

And because of MAPREDUCE-6302, it is possible that MR AM cancel all reducer requests and re-add
all reducer requests at the same heartbeat, so the #containers increases fast in RM side and
finally it becomes a enormous number. Since MAPREDUCE-6302 is included by branch-2.6/2.7,
we need to back port this patch to branch-2.6/2.7 as well.


> Job hangs as ask is not updated after ramping down of all reducers
> ------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6514
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6514
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>            Priority: Blocker
>         Attachments: MAPREDUCE-6514.01.patch, MAPREDUCE-6514.02.patch
>
>
> In RMContainerAllocator#preemptReducesIfNeeded, we simply clear the scheduled reduces
map and put these reducers to pending. This is not updated in ask. So RM keeps on assigning
and AM is not able to assign as no reducer is scheduled(check logs below the code).
> If this is updated immediately, RM will be able to schedule mappers immediately which
anyways is the intention when we ramp down reducers.
> Scheduler need not allocate for ramped down reducers
> This if not handled can lead to map starvation as pointed out in MAPREDUCE-6513
> {code}
>  LOG.info("Ramping down all scheduled reduces:"
>             + scheduledRequests.reduces.size());
>         for (ContainerRequest req : scheduledRequests.reduces.values()) {
>           pendingReduces.add(req);
>         }
>         scheduledRequests.reduces.clear();
> {code}
> {noformat}
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Container not assigned : container_1437451211867_1485_01_000215
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Cannot assign container Container: [ContainerId: container_1437451211867_1485_01_000216, NodeId:
hdszzdcxdat6g06u04p:26009, NodeHttpAddress: hdszzdcxdat6g06u04p:26010, Resource: <memory:4096,
vCores:1>, Priority: 10, Token: Token { kind: ContainerToken, service: 10.2.33.236:26009
}, ] for a reduce as either  container memory less than required 4096 or no pending reduce
tasks - reduces.isEmpty=true
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Container not assigned : container_1437451211867_1485_01_000216
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Cannot assign container Container: [ContainerId: container_1437451211867_1485_01_000217, NodeId:
hdszzdcxdat6g06u06p:26009, NodeHttpAddress: hdszzdcxdat6g06u06p:26010, Resource: <memory:4096,
vCores:1>, Priority: 10, Token: Token { kind: ContainerToken, service: 10.2.33.239:26009
}, ] for a reduce as either  container memory less than required 4096 or no pending reduce
tasks - reduces.isEmpty=true
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message