hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
Date Fri, 15 Sep 2017 19:46:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168420#comment-16168420

Daniel Templeton commented on YARN-2162:

My comments:
# In {{AllocationFileLoaderService.loadQueue()}}, instead of removing the max > min check,
can we make it conditional?  Use {{fitsIn()}} if they're both absolute and numerical comparison
if they're both percentages, and ignore the check otherwise, maybe with a warning.  Similar
in {{AllocationConfiguration.getMaxResources()}}. Even though {{FSQueue.getMaxShare()}} does
the check, it would be nice to have the warning earlier while you still have the configuration
details to put in the message.
# Looks like {{AllocationConfiguration.DEFAULT_CALCULATOR}} is unused.  Not your fault, so
you can file a newbie JIRA if you prefer.
# In {{AllocationConfiguration.initFSQueue()}}, you should probably change all the metrics
inits to use {{queue.get...()}}.
# In the tests, why do we need to set the calculator on the mock?
# It seems like we should use "vcores" instead of "cpu" for consistency.
# In {{FairSchedulerConfiguration.findPercentage()}}, you can probably drop the trailing while
space in the pattern. You also don't need the group 2 parentheses.  And the first '*' can
probably be a '+', but you might have done that intentionally.
# In {{TestFairSchedulerConfiguration.testParseResourceConfigValue()}}, you're missing some
negative tests, like one percentage and one absolute.
# Even though you have the tests in {{testParseResourceConfigValue()}}, it would probably
be a good idea to add some tests for {{ResourceConfiguration}} itself.
# Have you considered performance impact?  Every call to {{getMaxShare()}} causes a read lock
to be locked and unlocked and clones a resource.  That sounds kinda heavy to me for how often
it gets called.  Probably worth testing.

> add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
> -----------------------------------------------------------------------------------------
>                 Key: YARN-2162
>                 URL: https://issues.apache.org/jira/browse/YARN-2162
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler, scheduler
>            Reporter: Ashwin Shankar
>            Assignee: Yufei Gu
>              Labels: scheduler
>         Attachments: YARN-2162.001.patch, YARN-2162.002.patch, YARN-2162.003.patch, YARN-2162.004.patch,
> minResources and maxResources in fair scheduler configs are expressed in terms of absolute
numbers X mb, Y vcores. 
> As a result, when we expand or shrink our hadoop cluster, we need to recalculate and
change minResources/maxResources accordingly, which is pretty inconvenient.
> We can circumvent this problem if we can optionally configure these properties in terms
of percentage of cluster capacity. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message