hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-326) Add multi-resource scheduling to the fair scheduler
Date Thu, 25 Apr 2013 23:26:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642370#comment-13642370
] 

Sandy Ryza commented on YARN-326:
---------------------------------

Thanks for taking a look Andrew.  Minimum shares exceeding the cluster size and particularly
the cluster size when nodes are down is definitely a concern.  However, I'm wary of diverging
from how it's been configured in the past, which is in terms of absolute amounts.  Currently,
nothing crazy will happen in this situation either, resources will just go to the apps that
are the farthest behind their fair shares.  

Also, I think it's reasonable that minimum shares should be able to be different amounts for
different resources.  Suppose the geology department, which requires a bunch of cores, and
the history department, which runs jobs that need a lot of memory, decide to get together
on a cluster.  I think we should be able to give the geology department lots of CPU without
giving it lots of RAM.
                
> Add multi-resource scheduling to the fair scheduler
> ---------------------------------------------------
>
>                 Key: YARN-326
>                 URL: https://issues.apache.org/jira/browse/YARN-326
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>    Affects Versions: 2.0.2-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: FairSchedulerDRFDesignDoc.pdf, YARN-326.patch
>
>
> With YARN-2 in, the capacity scheduler has the ability to schedule based on multiple
resources, using dominant resource fairness.  The fair scheduler should be able to do multiple
resource scheduling as well, also using dominant resource fairness.
> More details to come on how the corner cases with fair scheduler configs such as min
and max resources will be handled.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message