hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anubhav Dhoot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3676) Disregard 'assignMultiple' directive while scheduling apps with NODE_LOCAL resource requests
Date Fri, 04 Sep 2015 18:50:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731244#comment-14731244
] 

Anubhav Dhoot commented on YARN-3676:
-------------------------------------

Thanks [~asuresh] for working on this. I see the patch continues assigning on the node if
you have *any* app which has a specific request on that node. But the scheduling attempt (via
queueMgr.getRootQueue().assignContainer(node)) does not restrict which apps will get allocation
on that node. So one could end up assigning the next container on the node for an app which
may not have a specific request for that node. 
I see two choices. 
a) Smaller change - We should allow subsequent assignments only for the node local only apps?
And you already have that list in the map. That can end up prioritizing the application's
node local request over other applications.
b) BIgger change - Once we have picked the app based on priority, we allow it to assign multiple
containers if there are multiple node local requests for that node.
 

> Disregard 'assignMultiple' directive while scheduling apps with NODE_LOCAL resource requests
> --------------------------------------------------------------------------------------------
>
>                 Key: YARN-3676
>                 URL: https://issues.apache.org/jira/browse/YARN-3676
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>         Attachments: YARN-3676.1.patch, YARN-3676.2.patch, YARN-3676.3.patch, YARN-3676.4.patch,
YARN-3676.5.patch
>
>
> AssignMultiple is generally set to false to prevent overloading a Node (for eg, new NMs
that have just joined)
> A possible scheduling optimization would be to disregard this directive for apps whose
allowed locality is NODE_LOCAL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message