hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6514) Update ask to indicate to RM that it need not allocate for ramped down reducers
Date Sat, 17 Oct 2015 09:16:05 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961823#comment-14961823

Sunil G commented on MAPREDUCE-6514:

Yes [~rohithsharma], thanks for the detailed analysis. This improvement can definitely avoid
many unnecessary allocations from RM side.
And if any extra container allocation happened from RM side within a heartbeat interval (AM
requested less number of reducers, however more containers were already allocated by RM),
such containers has to be rejected by AM and we already do that. I hope this number wont be
big as it may happen only within a heartbeat interval.

> Update ask to indicate to RM that it need not allocate for ramped down reducers
> -------------------------------------------------------------------------------
>                 Key: MAPREDUCE-6514
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6514
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.7.1
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
> In RMContainerAllocator#preemptReducesIfNeeded, we simply clear the scheduled reduces
map and put these reducers to pending. This is not updated in ask. So RM keeps on assigning
and AM is not able to assign as no reducer is scheduled(check logs below the code).
> If this is updated immediately, RM will be able to schedule mappers immediately which
anyways is the intention when we ramp down reducers.
> {code}
>  LOG.info("Ramping down all scheduled reduces:"
>             + scheduledRequests.reduces.size());
>         for (ContainerRequest req : scheduledRequests.reduces.values()) {
>           pendingReduces.add(req);
>         }
>         scheduledRequests.reduces.clear();
> {code}

This message was sent by Atlassian JIRA

View raw message