hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1042) add ability to specify affinity/anti-affinity in container requests
Date Mon, 08 Jun 2015 11:59:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577079#comment-14577079
] 

Steve Loughran commented on YARN-1042:
--------------------------------------

thanks for this. I'm afraid I don't know enough about the YARN scheduling to review it properly
—hopefully the experts will.

meanwhile, 
# does this let us specify different rules for different container requests? The uses we see
in slider do differentiate different requests (e.g. we care more about affinity of hbase masters
than we do about workers). 
# how would we set up the enum for if we ever want to add different placement policies in
future? I think the strategy would be to have a "no-preferences" policy, which would be the
default, and different from affinity (==I really want them on the same node) and anti-affinity.
Then we could have a {{switch}} statement to choose placement, rather than just an {{isAffinity()}}
predicate.
# hadoop's [code rules|https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md#code-style]
are "two spaces, no tabs". 

> add ability to specify affinity/anti-affinity in container requests
> -------------------------------------------------------------------
>
>                 Key: YARN-1042
>                 URL: https://issues.apache.org/jira/browse/YARN-1042
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Arun C Murthy
>         Attachments: YARN-1042-001.patch, YARN-1042-demo.patch
>
>
> container requests to the AM should be able to request anti-affinity to ensure that things
like Region Servers don't come up on the same failure zones. 
> Similarly, you may be able to want to specify affinity to same host or rack without specifying
which specific host/rack. Example: bringing up a small giraph cluster in a large YARN cluster
would benefit from having the processes in the same rack purely for bandwidth reasons.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message