hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable
Date Mon, 18 Apr 2016 06:15:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245187#comment-15245187
] 

Rohith Sharma K S commented on YARN-4963:
-----------------------------------------

Thanks [~nroberts] for initiating discussion on this. We have seen off_switch assignment issue
in large cluster as you described.  Especially when cluster is running fully occupied resource,
and container release happens all together from one node, only 1 container is assigned this
node. This makes user thinks that why assignment is not happening even though resource is
free. 
IMO, I think  application specific configurations should be there rather at scheduler level.
Some applications are fine with assigning containers in off_switch they can specify number
of containers to be assigned. But few applications are very strict to node locality, they
can configure 1 in off_switch.
Thoughts?

> capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-4963
>                 URL: https://issues.apache.org/jira/browse/YARN-4963
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>    Affects Versions: 3.0.0, 2.7.2
>            Reporter: Nathan Roberts
>            Assignee: Nathan Roberts
>         Attachments: YARN-4963.001.patch
>
>
> Currently the capacity scheduler will allow exactly 1 OFF_SWITCH assignment per heartbeat.
With more and more non MapReduce workloads coming along, the degree of locality is declining,
causing scheduling to be significantly slower. It's still important to limit the number of
OFF_SWITCH assignments to avoid densely packing OFF_SWITCH containers onto nodes. 
> Proposal is to add a simple config that makes the number of OFF_SWITCH assignments configurable.
> Will upload candidate patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message