hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable
Date Fri, 22 Apr 2016 13:20:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253902#comment-15253902
] 

Naganarasimha G R commented on YARN-4963:
-----------------------------------------

Thanks for the clarification [~wangda] & [~nroberts],  yes point 2 addresses the same
issue and my mistake i missed to read this. And also agree to the focus of this jira to be
specific to the system level OFF-SWITCH configuration.

bq. so I think when we do the application-level support the default would need to be either
unlimited or some high value, otherwise we force all applications to set this limit to something
other than 1 to get decent OFF_SWITCH scheduling behavior.
Once we have system level OFF-SWITCH configuration do we require app level default also ?
IIUC by default we try to make use of system level OFF-SWITCH configuration unless explicitly
overridden by the app (implementation can be further discussed in that jira)
bq. Sure, my application scheduled very quickly but my locality was terrible so I caused a
lot of unnecessary cross-switch traffic. So I think we'll need some system-minimums that will
prevent this type of abuse.
This point is debatable, even though i agree your point for controlling cross-switch traffic,
but still the app is performing under its capacity limits so would it be good to limit it
control it.
bq. If application A meets its OFF-SWITCH-per-node limit, do we offer the node to other applications
in the same queue?
any limitations if we offer the node to other applications in the same queue ? it should be
fine right ?





> capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-4963
>                 URL: https://issues.apache.org/jira/browse/YARN-4963
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>    Affects Versions: 3.0.0, 2.7.2
>            Reporter: Nathan Roberts
>            Assignee: Nathan Roberts
>         Attachments: YARN-4963.001.patch
>
>
> Currently the capacity scheduler will allow exactly 1 OFF_SWITCH assignment per heartbeat.
With more and more non MapReduce workloads coming along, the degree of locality is declining,
causing scheduling to be significantly slower. It's still important to limit the number of
OFF_SWITCH assignments to avoid densely packing OFF_SWITCH containers onto nodes. 
> Proposal is to add a simple config that makes the number of OFF_SWITCH assignments configurable.
> Will upload candidate patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message