hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Yan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1010) FairScheduler: decouple container scheduling from nodemanager heartbeats
Date Mon, 30 Sep 2013 23:47:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782437#comment-13782437

Wei Yan commented on YARN-1010:

Updates in the patch.

(1) The {{FairScheduler}} launches a thread to do the continuous scheduler.
(2) Several configuration fields:
{{yarn.scheduler.fair.continuous.scheduling.enabled}}. Whether to enable continuous scheduling.
The default value is false.
{{yarn.scheduler.fair.continuous.scheduling.sleep.time.ms}}. The sleep time for each round
of continuous scheduling, default valus is 5 ms.

Configurations for delay scheduling:
{{yarn.scheduler.fair.locality.threshold.node.time.ms}}. Time threshold for node locality.
The default value is -1L.
{{yarn.scheduler.fair.locality.threshold.rack.time.ms}}. Time threshold for rack locality.
The default value is -1L.
(3) Add test cases for continuous scheduling in {{TestFairScheduler}}, and the delay scheduling
mechanism in {{TestFSSchedulerApp}}.

> FairScheduler: decouple container scheduling from nodemanager heartbeats
> ------------------------------------------------------------------------
>                 Key: YARN-1010
>                 URL: https://issues.apache.org/jira/browse/YARN-1010
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 2.1.0-beta
>            Reporter: Alejandro Abdelnur
>            Assignee: Wei Yan
>            Priority: Critical
>         Attachments: YARN-1010.patch
> Currently scheduling for a node is done when a node heartbeats.
> For large cluster where the heartbeat interval is set to several seconds this delays
scheduling of incoming allocations significantly.
> We could have a continuous loop scanning all nodes and doing scheduling. If there is
availability AMs will get the allocation in the next heartbeat after the one that placed the

This message was sent by Atlassian JIRA

View raw message