hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sujeet Varakhedi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2877) Extend YARN to support distributed scheduling
Date Fri, 21 Nov 2014 19:26:36 GMT

    [ https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221309#comment-14221309
] 

Sujeet Varakhedi commented on YARN-2877:
----------------------------------------

+ 1 for distributed scheduling and SQL engines for Hadoop can greatly benefit from it. We
also need to look at a design we can give AMs more control over scheduling policies where
RM just acts a source of overall cluster state, NM's have local queues and then based on NM
queue wait times AM's can decide where to requests tasks. Similar to how Sparrow works. This
kind of scheduling becomes important for services that need dedicated non-shared clusters
like HBASE and HAWQ.

> Extend YARN to support distributed scheduling
> ---------------------------------------------
>
>                 Key: YARN-2877
>                 URL: https://issues.apache.org/jira/browse/YARN-2877
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Sriram Rao
>
> This is an umbrella JIRA that proposes to extend YARN to support distributed scheduling.
 Briefly, some of the motivations for distributed scheduling are the following:
> 1. Improve cluster utilization by opportunistically executing tasks otherwise idle resources
on individual machines.
> 2. Reduce allocation latency.  Tasks where the scheduling time dominates (i.e., task
execution time is much less compared to the time required for obtaining a container from the
RM).
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message