hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7391) Consider square root instead of natural log for size-based weight
Date Wed, 25 Oct 2017 18:26:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219241#comment-16219241
] 

Yufei Gu commented on YARN-7391:
--------------------------------

We don't want app weight grow too fast or too slow. How do we define too fast or too slow?
It's mainly based on the size of demand and expectation of users. Agreed with Daniel, there
is no answer without an analysis of user cases.

It is totally unnecessary to lock the scheduler only for {{getWeights()}} from an app. It
would be a performance issue while # of app is large. We could definitely do something to
improve.

> Consider square root instead of natural log for size-based weight
> -----------------------------------------------------------------
>
>                 Key: YARN-7391
>                 URL: https://issues.apache.org/jira/browse/YARN-7391
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 3.0.0-beta1
>            Reporter: Steven Rand
>
> Currently for size-based weight, we compute the weight of an app using this code from
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java#L377:
> {code}
>       if (sizeBasedWeight) {
>         // Set weight based on current memory demand
>         weight = Math.log1p(app.getDemand().getMemorySize()) / Math.log(2);
>       }
> {code}
> Because the natural log function grows slowly, the weights of two apps with hugely different
memory demands can be quite similar. For example, {{weight}} evaluates to 14.3 for an app
with a demand of 20 GB, and evaluates to 19.9 for an app with a demand of 1000 GB. The app
with the much larger demand will still have a higher weight, but not by a large amount relative
to the sum of those weights.
> I think it's worth considering a switch to a square root function, which will grow more
quickly. In the above example, the app with a demand of 20 GB now has a weight of 143, while
the app with a demand of 1000 GB now has a weight of 1012. These weights seem more reasonable
relative to each other given the difference in demand between the two apps.
> The above example is admittedly a bit extreme, but I believe that a square root function
would also produce reasonable results in general.
> The code I have in mind would look something like:
> {code}
>       if (sizeBasedWeight) {
>         // Set weight based on current memory demand
>         weight = Math.sqrt(app.getDemand().getMemorySize());
>       }
> {code}
> Would people be comfortable with this change?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message