hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-326) Add multi-resource scheduling to the fair scheduler
Date Fri, 26 Apr 2013 19:50:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643176#comment-13643176
] 

Sandy Ryza commented on YARN-326:
---------------------------------

What I meant is that we may want to limit the geology department to low memory/high CPU jobs,
because we bought all the memory for the history department in the first place, and geology
shouldn't be able to then steal it.  That said, I don't really know how realistic a scenario
this is.

Thinking about this more, though, a problem with my proposal is that an entity with a min
share in one resource can get an unlimited amount of other resources against a queue with
no min share.  E.g. if I have a min share of 10 CPUs, and you don't have any min share, I
can get all the memory in the cluster as long as I stay under 10 CPUs.  This issue would not
exist in your world, as eventually memory would become my dominant resource.

Because of that, I think the fractional minimum dominant share is probably the right way to
go.  We still need to maintain backwards compatibility though, so we might have to do some
sort of hybrid.
                
> Add multi-resource scheduling to the fair scheduler
> ---------------------------------------------------
>
>                 Key: YARN-326
>                 URL: https://issues.apache.org/jira/browse/YARN-326
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>    Affects Versions: 2.0.2-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: FairSchedulerDRFDesignDoc.pdf, YARN-326.patch, YARN-326.patch
>
>
> With YARN-2 in, the capacity scheduler has the ability to schedule based on multiple
resources, using dominant resource fairness.  The fair scheduler should be able to do multiple
resource scheduling as well, also using dominant resource fairness.
> More details to come on how the corner cases with fair scheduler configs such as min
and max resources will be handled.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message