hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1042) add ability to specify affinity/anti-affinity in container requests
Date Wed, 27 May 2015 11:08:22 GMT

    [ https://issues.apache.org/jira/browse/YARN-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560776#comment-14560776

Weiwei Yang commented on YARN-1042:

I am thinking about following approach, appreciate for suggestions : )

In ApplicationSubmissionContext class, add a new argument to indicate the container allocation
rule in terms of affinity/anti-affinity. RM will follow the certain rules to allocate containers
for this application. The argument is an instance of class ContainerAllocationRule(new), this
class defines several types allocation rule, such as 

 * AFFINITY_REQUIRED: containers MUST be allocated on the same host/rack
 * AFFINITY_PREFERED: prefer to allocate containers on same host/rack if possible
 * ANTI_AFFINITY_REQUIRED: containers MUST be allocated on different hosts/racks
 * ANTI_AFFINITY_PREFERED: prefer to allocate containers on different hosts/racks if possible

Each of these rules will have a handler on the RM side to add some control on container allocation.
When a client submits an application with a certain ContainerAllocationRule to RM, this information
will be added into ApplicationAttemptId (because the allocation rule is defined per application),
when RM uses registered scheduler to allocate containers, it can retrieve the rule from ApplicationAttemptId
and call particular handler during the allocation. The code can be added into SchedulerApplicationAttempt.pullNewlyAllocatedContainersAndNMTokens
so to avoid modifying all schedulers.

> add ability to specify affinity/anti-affinity in container requests
> -------------------------------------------------------------------
>                 Key: YARN-1042
>                 URL: https://issues.apache.org/jira/browse/YARN-1042
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Arun C Murthy
>         Attachments: YARN-1042-demo.patch
> container requests to the AM should be able to request anti-affinity to ensure that things
like Region Servers don't come up on the same failure zones. 
> Similarly, you may be able to want to specify affinity to same host or rack without specifying
which specific host/rack. Example: bringing up a small giraph cluster in a large YARN cluster
would benefit from having the processes in the same rack purely for bandwidth reasons.

This message was sent by Atlassian JIRA

View raw message