hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-9116) Capacity Scheduler: implements queue level maximum-allocation inheritance
Date Mon, 21 Jan 2019 05:13:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16747659#comment-16747659

Weiwei Yang commented on YARN-9116:

Hi [~aihuaxu]

Sorry for the late reply, I was on a trip.

The timeout issue should not be related to this patch, I saw it in different jenkins results
too. No worries.
{quote} if the configuration for some queues is incorrect or larger than the cluster level,
exception is also thrown to let the admin to correct it.
Yes, this is desired behavior. We should let it fast-fail instead of altering the value under-cover.

Latest patch looks almost good to me, few things

1). CapacityConfiguration.java
long queueMemory = csConf.getQueueMaximumAllocationMb(queue);
if we are expecting {{long}} value for queue max memory allocation, can we make sure {{getQueueMaximumAllocationMb}}
returns {{long}} instead of {{int}}?

2) TestCapacityScheduler#testQueuesMaxAllocationInheritance

Right now this UT covers the basic inheritance of queue-max. To improve the coverage, can
we set something like:
root.b -> 7168
root.b2 -> 1024
then verify values for queue: root.b, root.b.b1, root.b.b2.

3) can we add a negative UT, by setting a queue-max bigger than global max, in order to verify
we can get a proper failure in such case.

4) There are some checkstyle issues in latest patch, would you mind to fix them?


> Capacity Scheduler: implements queue level maximum-allocation inheritance
> -------------------------------------------------------------------------
>                 Key: YARN-9116
>                 URL: https://issues.apache.org/jira/browse/YARN-9116
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>    Affects Versions: 2.7.0
>            Reporter: Aihua Xu
>            Assignee: Aihua Xu
>            Priority: Major
>         Attachments: YARN-9116.1.patch, YARN-9116.2.patch, YARN-9116.3.patch
> YARN-1582 adds the support of maximum-allocation-mb configuration per queue which is
targeting to support larger container features on dedicated queues (larger maximum-allocation-mb/maximum-allocation-vcores
for such queue) . While to achieve larger container configuration, we need to increase the
global maximum-allocation-mb/maximum-allocation-vcores (e.g. 120G/256) and then override those
configurations with desired values on the queues since queue configuration can't be larger
than cluster configuration. There are many queues in the system and if we forget to configure
such values when adding a new queue, then such queue gets default 120G/256 which typically
is not what we want.  
> We can come up with a queue-default configuration (set to normal queue configuration
like 16G/8), so the leaf queues gets such values by default.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message