hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5479) FairScheduler: Scheduling performance improvement
Date Tue, 09 Aug 2016 14:52:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413653#comment-15413653
] 

Jason Lowe commented on YARN-5479:
----------------------------------

bq. While doing so does not seemly cause any problem in production (fairness is slightly damaged
locally, but within acceptable range.

What is acceptable for your production may not be acceptable to others.  We're changing the
requirements, and that could have ramifications for some users.  It's hard to say, which is
why I'd rather avoid going there unless absolutely necessary.

bq. Shall we make this issue an umbrella?

Yep, seems an appropriate place to gather performance improvements, although as mentioned
above some of these may not be (or should not be) specific to the FairScheduler.

> FairScheduler: Scheduling performance improvement
> -------------------------------------------------
>
>                 Key: YARN-5479
>                 URL: https://issues.apache.org/jira/browse/YARN-5479
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler, resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: He Tianyi
>            Assignee: He Tianyi
>
> Currently ResourceManager uses a single thread to handle async events for scheduling.
As number of nodes grows, more events need to be processed in time in FairScheduler. Also,
increased number of applications & queues slows down processing of each single event.

> There are two cases that slow processing of nodeUpdate events is problematic:
> A. global throughput is lower than number of nodes through heartbeat rounds. This keeps
resource from being allocated since the inefficiency.
> B. global throughput meets the need, but for some of these rounds, events of some nodes
cannot get processed before next heartbeat. This brings inefficiency handling burst requests
(i.e. newly submitted MapReduce application cannot get its all task launched soon given enough
resource).
> Pretty sure some people will encounter the problem eventually after a single cluster
is scaled to several K of nodes (even with {{assignmultiple}} enabled).
> This issue proposes to perform several optimization towards performance in FairScheduler
{{nodeUpdate}} method. To be specific:
> A. trading off fairness with efficiency, queue & app sorting can be skipped (or should
this be called 'delayed sorting'?). we can either start another dedicated thread to do the
sorting & updating, or actually perform sorting after current result have been used several
times (say sort once in every 100 calls.)
> B. performing calculation on {{Resource}} instances is expensive, since at least 2 objects
({{ResourceImpl}} and its proto builder) is created each time (using 'immutable' apis). the
overhead can be eliminated with a light-weighted implementation of Resource, which do not
instantiate a builder until necessary, because most instances are used as intermediate result
in scheduler instead of being exchanged via IPC. Also, {{createResource}} is using reflection,
which can be replaced by a plain {{new}} (for scheduler usage only). furthermore, perhaps
we could 'intern' resource to avoid allocation.
> C. other minor changes: such as move {{updateRootMetrics}} call to {{update}}, making
root queue metrics eventual consistent (which may satisfies most of the needs). or introduce
counters to {{getResourceUsage}} and make changing of resource incrementally instead of recalculate
each time.
> With A and B, I was looking at 4 times improvement in a cluster with 2K nodes.
> Suggestions? Opinions?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message