hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
Date Tue, 21 Apr 2015 20:21:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505678#comment-14505678

Wangda Tan commented on YARN-3319:

Some minor comments about configuration part: 
1) getting config mapping should be a generic method, you can use Configuration.iterator()
to get all properties of a policy.
2) Policy will handle properties such as "size-based-weight" in its own configure(..)
3) Do you think is it better to make property in <queue-name>.ordering-policy.<policy-name>.<property-key>?
In your example, "sizeBasedWeight" more like a property of fair policy instead of others.
I perfer add a <policy-name>
4) "sizeBasedWeight" to "size-based-weight-enabled" (or enable-size-based-weight) to make
it consistent with other configurations of CS. Add the enable to indicate it's a boolean instead
of float.
5) Make the String="size-based-weight" to a final static variable like other configuration
keys. Same of "fifo" and "fifo". Test should use these final static variables as well

1) Mark get/setSizeBasedWeight to VisibleForTesting?
2) getMagnitude should be private
3) LOG is not used by anybody, could be removed.

Findbugs warning?

> Implement a FairOrderingPolicy
> ------------------------------
>                 Key: YARN-3319
>                 URL: https://issues.apache.org/jira/browse/YARN-3319
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Craig Welch
>            Assignee: Craig Welch
>         Attachments: YARN-3319.13.patch, YARN-3319.14.patch, YARN-3319.17.patch, YARN-3319.35.patch,
YARN-3319.39.patch, YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, YARN-3319.58.patch,
YARN-3319.70.patch, YARN-3319.71.patch, YARN-3319.72.patch, YARN-3319.73.patch
> Implement a FairOrderingPolicy which prefers to allocate to SchedulerProcesses with least
current usage, very similar to the FairScheduler's FairSharePolicy.  
> The Policy will offer allocations to applications in a queue in order of least resources
used, and preempt applications in reverse order (from most resources used). This will include
conditional support for sizeBasedWeight style adjustment
> Optionally, based on a conditional configuration to enable sizeBasedWeight (default false),
an adjustment to boost larger applications (to offset the natural preference for smaller applications)
will adjust the resource usage value based on demand, dividing it by the below value:
> Math.log1p(app memory demand) / Math.log(2);
> In cases where the above is indeterminate (two applications are equal after this comparison),
behavior falls back to comparison based on the application id, which is generally lexically
FIFO for that comparison

This message was sent by Atlassian JIRA

View raw message