hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (Jira)" <j...@apache.org>
Subject [jira] [Created] (YARN-10496) [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
Date Thu, 19 Nov 2020 18:42:00 GMT
Wangda Tan created YARN-10496:

             Summary: [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
                 Key: YARN-10496
                 URL: https://issues.apache.org/jira/browse/YARN-10496
             Project: Hadoop YARN
          Issue Type: New Feature
          Components: capacity scheduler
            Reporter: Wangda Tan

CapacityScheduler today doesn’t support an auto queue creation which is flexible enough.
The current constraints: 
 * Only leaf queues can be auto-created
 * A parent can only have either static queues or dynamic ones. This causes multiple constraints.
For example:

 * It isn’t possible to have a VIP user like Alice with a static queue root.user.alice with
50% capacity while the other user queues (under root.user) are created dynamically and they
share the remaining 50% of resources.

 * In comparison, FairScheduler allows the following scenarios, Capacity Scheduler doesn’t:
 ** This implies that there is no possibility to have both dynamically created and static
queues at the same time under root
 * A new queue needs to be created under an existing parent, while the parent already has
static queues
 * Nested queue mapping policy, like in the following example: 

|<rule name="nestedUserQueue" create=”true”>
        <rule name="primaryGroup" create="true" />
 * Here two levels of queues may need to be created 

If an application belongs to user _alice_ (who has the primary_group of _engineering_), the
scheduler checks whether _root.engineering_ exists, if it doesn’t,  it’ll be created.
Then scheduler checks whether _root.engineering.alice_ exists, and creates it if it doesn't.


When we try to move users from FairScheduler to CapacityScheduler, we face feature gaps which
blocks users migrate from FS to CS.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message