hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aihua Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-9116) Capacity Scheduler: add the default maximum-allocation-mb and maximum-allocation-vcores for the queues
Date Wed, 02 Jan 2019 18:26:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732283#comment-16732283

Aihua Xu commented on YARN-9116:

Thanks [~cheersyang] for the comment. Happy new year.

So you are suggesting the following, is that correct? Actually that would introduce many queue
level configuration if we don't introduce new property even with such inheritance. Even after
we implement inheritance mechanism, we have to set the global to be 120G/256vCores (the maximum
value allowed in the cluster) and then override all the top queues to be 16G/16vCores and
set the larger container top queue to 120G/256vCores. I feel the current approach is simpler
and straightforward. Let me know if you think the inheritance implementation is still needed,
but seems we do need to add additional configuration.

Queue level max inherits the value from its parent if it is not explicitly set
If queue level max is set explicitly, then it is honored without considering its parents

> Capacity Scheduler: add the default maximum-allocation-mb and maximum-allocation-vcores
for the queues
> ------------------------------------------------------------------------------------------------------
>                 Key: YARN-9116
>                 URL: https://issues.apache.org/jira/browse/YARN-9116
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>    Affects Versions: 2.7.0
>            Reporter: Aihua Xu
>            Assignee: Aihua Xu
>            Priority: Major
>         Attachments: YARN-9116.1.patch
> YARN-1582 adds the support of maximum-allocation-mb configuration per queue which is
targeting to support larger container features on dedicated queues (larger maximum-allocation-mb/maximum-allocation-vcores
for such queue) . While to achieve larger container configuration, we need to increase the
global maximum-allocation-mb/maximum-allocation-vcores (e.g. 120G/256) and then override those
configurations with desired values on the queues since queue configuration can't be larger
than cluster configuration. There are many queues in the system and if we forget to configure
such values when adding a new queue, then such queue gets default 120G/256 which typically
is not what we want.  
> We can come up with a queue-default configuration (set to normal queue configuration
like 16G/8), so the leaf queues gets such values by default.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message