hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6345) Add container tags to resource requests
Date Thu, 16 Mar 2017 02:09:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927330#comment-15927330

Vinod Kumar Vavilapalli commented on YARN-6345:

[~kkaranasos], glad to see movement on this!

I have two broad comments
 # We should add allocation-tags instead of container-tags. HBaseMaster can run in multiple
containers over its life-cycle. Yes, today allocation = container. But that is changing very
quickly - ref YARN-4726. Further, the placement request usually is "place me close to HBaseMaster",
not "place me close to this container-id" - a subtle but important difference.
 # API design: We should not add it to existing ResourceRequest object. We have a vision of
moving to a more modular structure for ResourceRequests - YARN-4902 gives a flavor of this.

More context below.

h4. (1) Allocation-tags
Allocation-tags are described in the design doc for YARN-4902.
Allocation-­tags is an option field usable by the scheduler and app developers to group allocations:
 - Every allocation can be associated with a limited number of allocation-­tags. Each allocation-­tag
is a key­-value pair of strings.
 - Applications can use an allocation­-tag to group and organize pending and satisfied allocations.
For example, a long­-running service can assign different role-­types as allocation-­tags
to different allocations, so that it can easily understand how much resources are assigned
to different roles.
 - Applications can also use target­-allocation­-tags as part of a placement-­strategy
described in the later sections. When an application refers to a target­-allocation-­tag
in the scheduling request, scheduler will use it to allocate resources for a ResourceRequest
respecting affinity/anti­-affinity to another ResourceRequest identified by the passed-­in
allocation­-tags. We will discuss more details about this in section 4.4 Allocation ­tags
are scoped under an application (application-­id). So, if app A wants to refer to app B’s
allocation­-tags as target for placement-­strategy (described later), app A must specify
both the application­-id of app B as well as the target tags. In the future, we can see if
there is a need to expand this scoping control to be user­name, cluster-­id etc.

h4. (2) API design
Today's ResourceRequest is already very messy. We already had to force fit Allocation ID in
semantics and structure (ref YARN-4888) to this. Adding tags as well as more (YARN-6348) to
this already brittle API will lead us to an evolutionary dead-end. Instead I propose that
we start creating a parallel structure of ResourceRequestCollection proposed in YARN-4902,
right under AllocateRequest record and start bridging from current ResourceRequest object
to the next generation object. We can let them both live and evolve in parallel by tying the
old and new structures via AllocationID.

> Add container tags to resource requests
> ---------------------------------------
>                 Key: YARN-6345
>                 URL: https://issues.apache.org/jira/browse/YARN-6345
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>            Reporter: Konstantinos Karanasos
> This JIRA introduces the notion of container tags.
> When an application submits container requests, it is allowed to attach to them a set
of string tags. The corresponding resource requests will also carry these tags.
> For example, a container that will be used for running an HBase Master can be marked
with the tag "hb-m". Another one belonging to a ZooKeeper application, can be marked as "zk".
> Through container tags, we will be able to express constraints that refer to containers
with the given tags.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message