hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aihua Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-9200) Enable resource configuration of queue capacity for different resources independently
Date Tue, 29 Jan 2019 18:52:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755286#comment-16755286

Aihua Xu commented on YARN-9200:

[~leftnoteasy], [~rohithsharma] and [~cheersyang] I'm trying to add the support to have separate
percentage value for each resource (i.e., have an array instead of float value for each capacity).
Seems there are much more change than I originally thought, especially the changes in QueueCapacities
and the related ones. Before I move forward with such massive changes, I attached the draft
and want to check with you folks if there is a better way. 

What I have done and I'm planning to do is: CapacitySchedulerConfiguration supports both "45"
or "memory=80,vCores=20" and internally keeps an array - one value per resource; in QueueCapacities
to map each label to an array of Capacities.

You feedback is appreciated.  

> Enable resource configuration of queue capacity for different resources independently
> -------------------------------------------------------------------------------------
>                 Key: YARN-9200
>                 URL: https://issues.apache.org/jira/browse/YARN-9200
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacity scheduler
>    Affects Versions: 3.1.0
>            Reporter: Aihua Xu
>            Assignee: Aihua Xu
>            Priority: Major
>         Attachments: YARN-9200.draft
> In capacity scheduler, currently two resource allocations are supported. 1. percentage
allocation for child queues -  the child queue gets a defined percentage of the resources
for all the resource types; 2. absolute values  (YARN-5881) - each resource is configured
an absolute values.
> Right now we can't mix these case together and it would also very confusing to mix them
in one cluster. The second case actually is more targeting toward cloud env. 
> In a non-cloud env, the ability to configure each resource independently is also useful,
but percentage is preferable over absolute value. One thought here is to add the percentage
configuration for each resource type on the queue. That would allow us to configure memory
bounded queues, or CPU bounded queues. We can also keep backward compatible: each resource
type just gets the same percentage if no percentage is configured for individual resource

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message