hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6050) AMs can't be scheduled on racks or nodes
Date Wed, 22 Mar 2017 01:49:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935642#comment-15935642

Robert Kanter commented on YARN-6050:

I looked into the {{TestCapacitySchedulerNodeLabelUpdate}} failure.  It looks like the problem
is that when {{RMServerUtils#getApplicableNodeCountForAM}} calls {{CommonNodeLabelsManager#getLabelsToNodes(RMNodeLabelsManager.NO_LABEL)}},
it always returns an empty list of {{NodeId}} because it's not a real label and it looks like
there's a bunch of special handling for it.  The original code that was here was calling {{CommonNodeLabelsManager#getActiveNMCountPerLabel(RMNodeLabelsManager.NO_LABEL)}}
(from the {{RMNodeLabelsManager}} subclass), which does return the number of nodes without
a label.  This behavior is inconsistent.... 

I'll try to figure out a way around this problem or some simple/cheap way to determine the
{{NodeId}} for unlabeled/default labeled nodes.  But from my quick look at the code and the
debugger, it doesn't look like this is tracked anywhere.

> AMs can't be scheduled on racks or nodes
> ----------------------------------------
>                 Key: YARN-6050
>                 URL: https://issues.apache.org/jira/browse/YARN-6050
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.9.0, 3.0.0-alpha2
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: YARN-6050.001.patch, YARN-6050.002.patch, YARN-6050.003.patch, YARN-6050.004.patch,
YARN-6050.005.patch, YARN-6050.006.patch, YARN-6050.007.patch, YARN-6050.008.patch, YARN-6050.009.patch,
YARN-6050.010.patch, YARN-6050.011.patch, YARN-6050.012.patch
> Yarn itself supports rack/node aware scheduling for AMs; however, there currently are
two problems:
> # To specify hard or soft rack/node requests, you have to specify more than one {{ResourceRequest}}.
 For example, if you want to schedule an AM only on "rackA", you have to create two {{ResourceRequest}},
like this:
> {code}
> ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
> ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS, true);
> {code}
> The problem is that the Yarn API doesn't actually allow you to specify more than one
{{ResourceRequest}} in the {{ApplicationSubmissionContext}}.  The current behavior is to either
build one from {{getResource}} or directly from {{getAMContainerResourceRequest}}, depending
on if {{getAMContainerResourceRequest}} is null or not.  We'll need to add a third method,
say {{getAMContainerResourceRequests}}, which takes a list of {{ResourceRequest}} so that
clients can specify the multiple resource requests.
> # There are some places where things are hardcoded to overwrite what the client specifies.
 These are pretty straightforward to fix.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message