Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 1 Jun 2017 05:10:04 +0000 (UTC)
From: "daemon (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13049967.1489134861000.338511.1496293804847@Atlassian.JIRA>
In-Reply-To: <JIRA.13049967.1489134861000@Atlassian.JIRA>
References: <JIRA.13049967.1489134861000@Atlassian.JIRA> <JIRA.13049967.1489134861024@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-6320) FairScheduler:Identifying apps to
 assign in updateThread
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 01 Jun 2017 05:10:10 -0000


    [ https://issues.apache.org/jira/browse/YARN-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032452#comment-16032452 ] 

daemon commented on YARN-6320:
------------------------------

I think I have solved the problem and in my test environment the scheduler can assign over 5000 containers per second.

> FairScheduler:Identifying apps to assign in updateThread
> --------------------------------------------------------
>
>                 Key: YARN-6320
>                 URL: https://issues.apache.org/jira/browse/YARN-6320
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Tao Jie
>
> In FairScheduler today, we have 1)UpdateThread that update queue/app status, fairshare, starvation info, 2)nodeUpdate triggered by NM heartbeat that do the scheduling. When we handle one nodeUpdate, we will top-down from the root queue to the leafqueues and find the most needy application to allocate container according to queue's fairshare. Also we should sort children at each hierarchy level.
> My thought is that we have a global sorted {{candidateAppList}} which keeps apps need to assign, and move the logic that "find app that should allocate resource to" from nodeUpdate to UpdateThread. In UpdateThread, we find candidate apps to assign and put them into {{candidateAppList}}. In nodeUpdate, we consume the list and allocate containers to apps. 
> As far as I see, we can have 3 benifits:
> 1, nodeUpdate() is invoked much more frequently than update() in UpdateThread, especially in a large cluster. As a result we can reduce much unnecessary sorting.
> 2, It will have better coordination with YARN-5829, we can indicate apps to assign more directly rather than let nodes find the best apps to assign.
> 3, It seems to be easier to introduce scheduling restricts such as nodelabel, affinity/anti-affinity into FS, since we can pre-allocate containers asynchronously.
> [~kasha], [~templedf], [~yufeigu] like to hear your thoughts.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org