hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1042) add ability to specify affinity/anti-affinity in container requests
Date Mon, 27 Jun 2016 23:50:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352072#comment-15352072

Wangda Tan commented on YARN-1042:

Hi [~cheersyang], thanks for the design doc, patch and ideas.

I think there're two major problems that we need to solve.
1) What the API should look like?
2) How to make it efficient enough?

For 1), I'm agree with most of the comments from [~cchen317], existing ResourceRequest API
has some flaws that we may need to think how to fix them. We have a design doc attached to
YARN-4902 with a lot of thoughts regarding to this, you can read it if you have interests.
And for anti-affinity/affinity API itself, we may need to consider anti-affinity/affinity
between applications/priorities. Sometimes different services can be launched by one single
YARN application, different roles from different services could possibly have different affinity/anti-affinity
requirements. And one service/job can affinity/anti-affinity to another service/job.

For 2), Existing scheduler is blindly looping all the nodes in the cluster, we need a global
view to schedule them for better performance, I have opened YARN-5139 for the global scheduling
framework and attached POC patch to it.

Currently I'm working on a prototype patch for this, I want to make this feature based on
YARN-5139 and probably based on YARN-4902.

I'm interested in taking this forward, assigning it to myself.

Would like to hear your thoughts.

> add ability to specify affinity/anti-affinity in container requests
> -------------------------------------------------------------------
>                 Key: YARN-1042
>                 URL: https://issues.apache.org/jira/browse/YARN-1042
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Steve Loughran
>            Assignee: Wangda Tan
>         Attachments: YARN-1042-demo.patch, YARN-1042-design-doc.pdf, YARN-1042.001.patch,
> container requests to the AM should be able to request anti-affinity to ensure that things
like Region Servers don't come up on the same failure zones. 
> Similarly, you may be able to want to specify affinity to same host or rack without specifying
which specific host/rack. Example: bringing up a small giraph cluster in a large YARN cluster
would benefit from having the processes in the same rack purely for bandwidth reasons.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message