hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6050) AMs can't be scheduled on racks or nodes
Date Tue, 14 Mar 2017 00:37:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923316#comment-15923316

Wangda Tan commented on YARN-6050:


bq. If you think it would be better to use not change the label manager code and instead filter
out the :0 nodes, I can also do that instead.
I'm OK with changes of RMNodeLabelsManager. But should we use concurrent set in RMNodeLabel?
Which can avoid copy of the whole set and it will be more consistent to latest set? (Copy
and read the Set cannot capture later changes of label-node mappings, but inconsistency result
only happens to concurrent Set when iterating and writing happen at the same time).

Some other comments:
1) RMAppManager#validateAndCreateResourceRequest, could you move ANY request to first of the
list? Since there're some assumptions in other logics like:
- {{RMAppImpl#getAmNodeLabelExpression}}: {{getAMResourceRequests().get(0).getNodeLabelExpression()}}
- {{RMServerUtils#getApplicableNodeCountForAM}}: {{amReqs.get(0).getNodeLabelExpression()}}

Rest of the patch looks good to me.

Considering size of the patch, I would prefer to have another set of eyes to look at it as
well. [~kasha]/[~asuresh].

> AMs can't be scheduled on racks or nodes
> ----------------------------------------
>                 Key: YARN-6050
>                 URL: https://issues.apache.org/jira/browse/YARN-6050
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.9.0, 3.0.0-alpha2
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: YARN-6050.001.patch, YARN-6050.002.patch, YARN-6050.003.patch, YARN-6050.004.patch,
YARN-6050.005.patch, YARN-6050.006.patch, YARN-6050.007.patch, YARN-6050.008.patch, YARN-6050.009.patch,
> Yarn itself supports rack/node aware scheduling for AMs; however, there currently are
two problems:
> # To specify hard or soft rack/node requests, you have to specify more than one {{ResourceRequest}}.
 For example, if you want to schedule an AM only on "rackA", you have to create two {{ResourceRequest}},
like this:
> {code}
> ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
> ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS, true);
> {code}
> The problem is that the Yarn API doesn't actually allow you to specify more than one
{{ResourceRequest}} in the {{ApplicationSubmissionContext}}.  The current behavior is to either
build one from {{getResource}} or directly from {{getAMContainerResourceRequest}}, depending
on if {{getAMContainerResourceRequest}} is null or not.  We'll need to add a third method,
say {{getAMContainerResourceRequests}}, which takes a list of {{ResourceRequest}} so that
clients can specify the multiple resource requests.
> # There are some places where things are hardcoded to overwrite what the client specifies.
 These are pretty straightforward to fix.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message