hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6514) Update ask to indicate to RM that it need not allocate for ramped down reducers
Date Sat, 17 Oct 2015 07:12:05 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961789#comment-14961789
] 

Rohith Sharma K S commented on MAPREDUCE-6514:
----------------------------------------------

Thanks [~varun_saxena] for creating issue. 

I too have seen many times this particular problem because of not updating ResourceRequest
when Ramping down the reducers. As a result, number of containers count in the ask is increased
as explained in the below

Consider MR job is running with 3 mapper and 100 reducers. All mappers were finished, and
3 reducers are running in the cluster.
Kill the task attempt of mapper. So Ramping down will happen.
{noformat}
2015-10-17 12:09:10,836 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Ramping down all scheduled reduces:97
{noformat}

Since resourcerequest is not decremented from remoteRequestsTable. Next time when ask for
97 reducers i.e *pendingReducers --> scheduledReducers*, which starts adding number of
containers to existing RR. So {{97(previous outstanding reducers)+ 100(total reducers)=197}}
{noformat}
2015-10-17 12:10:49,617 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
All maps assigned. Ramping up all remaining reduces:100
2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=98 #asks=2
2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=99 #asks=2
2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=100 #asks=2
......
......
2015-10-17 12:10:49,647 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=195 #asks=2
2015-10-17 12:10:49,648 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=196 #asks=2
2015-10-17 12:10:49,648 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=197 #asks=2
{noformat}

In ResourceManager.log
In the log, observe that *Containers: 100* before ramping down all the reducers. Afterwords,
MR ask for reducers requests, *Containers: 197*
{noformat}
2015-10-17 12:07:40,613 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
showRequests: application=application_1445063712158_0001 request={Priority: 10, Capability:
<memory:2048, vCores:1>, # Containers: 100, Location: *, Relax Locality: true, Node
Label Expression: }
{noformat}

After ramping down all the reducers and request for ask. 

{noformat}
2015-10-17 12:10:50,651 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
showRequests: application=application_1445063712158_0001 request={Priority: 10, Capability:
<memory:2048, vCores:1>, # Containers: 197, Location: *, Relax Locality: true, Node
Label Expression: }
{noformat}




Basically it causes unnecessary overhead on ResourceManager that for assignment and also effect
other applications container assignment. These allocated containers are rejected by MR.

> Update ask to indicate to RM that it need not allocate for ramped down reducers
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6514
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6514
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.7.1
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>
> In RMContainerAllocator#preemptReducesIfNeeded, we simply clear the scheduled reduces
map and put these reducers to pending. This is not updated in ask. So RM keeps on assigning
and AM is not able to assign as no reducer is scheduled(check logs below the code).
> If this is updated immediately, RM will be able to schedule mappers immediately which
anyways is the intention when we ramp down reducers.
> {code}
>  LOG.info("Ramping down all scheduled reduces:"
>             + scheduledRequests.reduces.size());
>         for (ContainerRequest req : scheduledRequests.reduces.values()) {
>           pendingReduces.add(req);
>         }
>         scheduledRequests.reduces.clear();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message