Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 9 Aug 2016 14:52:20 +0000 (UTC)
From: "Jason Lowe (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12995454.1470535847000.261839.1470754340560@Atlassian.JIRA>
In-Reply-To: <JIRA.12995454.1470535847000@Atlassian.JIRA>
References: <JIRA.12995454.1470535847000@Atlassian.JIRA> <JIRA.12995454.1470535847867@arcas>
Subject: [jira] [Commented] (YARN-5479) FairScheduler: Scheduling
 performance improvement
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 09 Aug 2016 14:52:23 -0000


    [ https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413653#comment-15413653 ] 

Jason Lowe commented on YARN-5479:
----------------------------------

bq. While doing so does not seemly cause any problem in production (fairness is slightly damaged locally, but within acceptable range.

What is acceptable for your production may not be acceptable to others.  We're changing the requirements, and that could have ramifications for some users.  It's hard to say, which is why I'd rather avoid going there unless absolutely necessary.

bq. Shall we make this issue an umbrella?

Yep, seems an appropriate place to gather performance improvements, although as mentioned above some of these may not be (or should not be) specific to the FairScheduler.

> FairScheduler: Scheduling performance improvement
> -------------------------------------------------
>
>                 Key: YARN-5479
>                 URL: https://issues.apache.org/jira/browse/YARN-5479
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler, resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: He Tianyi
>            Assignee: He Tianyi
>
> Currently ResourceManager uses a single thread to handle async events for scheduling. As number of nodes grows, more events need to be processed in time in FairScheduler. Also, increased number of applications & queues slows down processing of each single event. 
> There are two cases that slow processing of nodeUpdate events is problematic:
> A. global throughput is lower than number of nodes through heartbeat rounds. This keeps resource from being allocated since the inefficiency.
> B. global throughput meets the need, but for some of these rounds, events of some nodes cannot get processed before next heartbeat. This brings inefficiency handling burst requests (i.e. newly submitted MapReduce application cannot get its all task launched soon given enough resource).
> Pretty sure some people will encounter the problem eventually after a single cluster is scaled to several K of nodes (even with {{assignmultiple}} enabled).
> This issue proposes to perform several optimization towards performance in FairScheduler {{nodeUpdate}} method. To be specific:
> A. trading off fairness with efficiency, queue & app sorting can be skipped (or should this be called 'delayed sorting'?). we can either start another dedicated thread to do the sorting & updating, or actually perform sorting after current result have been used several times (say sort once in every 100 calls.)
> B. performing calculation on {{Resource}} instances is expensive, since at least 2 objects ({{ResourceImpl}} and its proto builder) is created each time (using 'immutable' apis). the overhead can be eliminated with a light-weighted implementation of Resource, which do not instantiate a builder until necessary, because most instances are used as intermediate result in scheduler instead of being exchanged via IPC. Also, {{createResource}} is using reflection, which can be replaced by a plain {{new}} (for scheduler usage only). furthermore, perhaps we could 'intern' resource to avoid allocation.
> C. other minor changes: such as move {{updateRootMetrics}} call to {{update}}, making root queue metrics eventual consistent (which may satisfies most of the needs). or introduce counters to {{getResourceUsage}} and make changing of resource incrementally instead of recalculate each time.
> With A and B, I was looking at 4 times improvement in a cluster with 2K nodes.
> Suggestions? Opinions?


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org