hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7655) avoid AM preemption caused by RRs for specific nodes or racks
Date Wed, 17 Jan 2018 02:11:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328166#comment-16328166
] 

Yufei Gu commented on YARN-7655:
--------------------------------

Do several AMs end up running on a limited number of NM? If the answer is Yes, what I am guessing
is that Spark's dynamic allocation and {{assignmultiple}} of FS work together to make that.
With Spark's dynamic allocation, an app may only have one AM container for a while before
it requires tasks containers(That's what I am guessing though, I'm not familiar with Spark).
Several AMs may be allocated into one or very few NMs while assignmultiple is on. This also
aligns with -1 maxAMShare in your setting.

If that so, YARN-1042(anti-affinity) would help, but it isn't done yet. The MAPREDUCE-6871
provide a hacky way to do that as well. You can specify nodes/racks for AMs, so you can spread
out AMs. This feature is not only for MR, Spark AM did the similar implementation. 

> avoid AM preemption caused by RRs for specific nodes or racks
> -------------------------------------------------------------
>
>                 Key: YARN-7655
>                 URL: https://issues.apache.org/jira/browse/YARN-7655
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 3.0.0
>            Reporter: Steven Rand
>            Assignee: Steven Rand
>            Priority: Major
>         Attachments: YARN-7655-001.patch
>
>
> We frequently see AM preemptions when {{starvedApp.getStarvedResourceRequests()}} in
{{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs that request containers
on a specific node. Since this causes us to only consider one node to preempt containers on,
the really good work that was done in YARN-5830 doesn't save us from AM preemption. Even though
there might be multiple nodes on which we could preempt enough non-AM containers to satisfy
the app's starvation, we often wind up preempting one or more AM containers on the single
node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM containers for an
RR that specifies a node or rack, then we should instead expand the search space to consider
all nodes. That way we take advantage of YARN-5830, and only preempt AMs if there's no alternative.
I've attached a patch with an initial implementation of this. We've been running it on a few
clusters, and have seen AM preemptions drop from double-digit occurrences on many days to
zero.
> Of course, the tradeoff is some loss of locality, since the starved app is less likely
to be allocated resources at the most specific locality level that it asked for. My opinion
is that this tradeoff is worth it, but interested to hear what others think as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message