hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6612) Update fair scheduler policies to be aware of resource types
Date Tue, 05 Sep 2017 21:38:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154362#comment-16154362

Yufei Gu commented on YARN-6612:

Thanks for the patch, [~templedf], it looks much better. 
- Zero-weight queues are supported by FS. We may only be lucky due to nobody uses it in DRF
scenario. People will tend to use DRF more in case of multiple resource types. In that sense,
it is necessary to handle zero-weight case.
- Let's talk offline about weight in min share calculation and sort by dominant min share
which is minShare/cluste.
- About naming, I think we have used the word 'share' for different concepts. 1. it has been
used in concepts like "Fair Share", "Min Share", "Max Share", which are not necessary but
usually absolute resource of a schedulable. 2. In DRF, 'share' refers to how big a fraction
of a resource is comparing to the cluster resource. In {{calculateShares}} and {{calculateMinShares}},
we don't only calculate how big a fraction of a used resource is comparing to the cluster
resource, but also calculate how much portion it has in Fair Share and Min Share. That's why
I think names of {{calculateShares}} and {{calculateMinShares}} are not ideal. In addition,
Java doc of those two functions should mention they calculate how much fair share/min share
the schedulable use.

> Update fair scheduler policies to be aware of resource types
> ------------------------------------------------------------
>                 Key: YARN-6612
>                 URL: https://issues.apache.org/jira/browse/YARN-6612
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler
>    Affects Versions: YARN-3926
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
>         Attachments: YARN-6612.YARN-3926.001.patch, YARN-6612.YARN-3926.002.patch, YARN-6612.YARN-3926.004.patch,
YARN-6612.YARN-3926.005.patch, YARN-6612.YARN-3926.006.patch, YARN-6612.YARN-3926.007.patch,

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message