hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2989) Better Load Balancing in Fair Scheduler
Date Tue, 23 Dec 2014 22:38:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257622#comment-14257622
] 

Karthik Kambatla commented on YARN-2989:
----------------------------------------

[~rkannan82] - as you might have noticed, we want to move towards continuous scheduling being
enabled by default. One shouldn't need assignMultiple with continuous scheduling. 

> Better Load Balancing in Fair Scheduler
> ---------------------------------------
>
>                 Key: YARN-2989
>                 URL: https://issues.apache.org/jira/browse/YARN-2989
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.5.0
>            Reporter: Kannan Rajah
>
> While porting Fair Scheduler from MR1, we seem to have changed the logic behind task
distribution across nodes (MAPREDUCE-3451).
> In MR1, a load factor was computed using runnableMaps/totalMapSlots and this was used
to determine how many tasks need to be given to a node such that the overall cluster load
is evenly distributed. In one heartbeat, we could assign multiple tasks. In YARN, we have
the option to assign multiple tasks to a node, but this is disabled by default (YARN-302).
Even when it is enabled, the number of tasks to assign is statically configured. So it won't
ensure that load is evenly distributed. Why not bring back the load factor based check? Any
reason why it was not done? This is actually more relevant with label based scheduling.
> If there are no objections, I would like to implement it for both normal and label based
scheduling scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message