hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Rand (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6956) preemption may only consider resource requests for one node
Date Sun, 06 Aug 2017 16:36:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115847#comment-16115847
] 

Steven Rand edited comment on YARN-6956 at 8/6/17 4:35 PM:
-----------------------------------------------------------

Hi [~daniel@cloudera.com], thanks for the quick reply and explanation. That concern definitely
makes sense, and in general YARN-6163 seems like a good change.

However, what I'm seeing is that only considering RRs for one node actually causes some of
my apps to remain starved for quite a long time. The series of events that happens in a loop
is:

1. The app is correctly considered to be starved
2. The app has many RRs, several of which can be satisfied, but only one RR is actually considered
for preemption as per this JIRA's description
3. That particular RR happens to be for a node on which no containers can be preempted for
the app, so the app remains starved

Since the order of the list of RRs is the same each time through the loop, the same RR is
always considered, no containers are preempted, and the app remains starved, even though it
has other RRs that could be satisfied.

I haven't thought enough yet about what a solution would look like, but it seems like we should
be able to keep the benefits of YARN-6163 while also avoiding this issue. I'll try to have
a patch within the next few days if people agree that we should change the behavior.


was (Author: steven rand):
Hi [~daniel@cloudera.com], thanks for the quick reply and explanation. That concern definitely
makes sense, and in general YARN-6163 seems like a good change.

However, what I'm seeing is that only considering RRs for one node actually causes some of
my apps to remain starved for quite a long time. The series of events that happens in a loop
is:

1. The app is correctly considered to be starved
2. The app has many RRs, several of which can be satisfied, but only one RR is actually considered
for preemption as per this JIRA's description
3. That particular RR happens to be for a node on which the no containers can be preempted
for the app, so the app remains starved

Since the order of the list of RRs is the same each time through the loop, the same RR is
always considered, no containers are preempted, and the app remains starved, even though it
has other RRs that could be satisfied.

I haven't thought enough yet about what a solution would look like, but it seems like we should
be able to keep the benefits of YARN-6163 while also avoiding this issue. I'll try to have
a patch within the next few days if people agree that we should change the behavior.

> preemption may only consider resource requests for one node
> -----------------------------------------------------------
>
>                 Key: YARN-6956
>                 URL: https://issues.apache.org/jira/browse/YARN-6956
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.9.0, 3.0.0-beta1
>         Environment: CDH 5.11.0
>            Reporter: Steven Rand
>
> I'm observing the following series of events on a CDH 5.11.0 cluster, which seem to be
possible after YARN-6163:
> 1. An application is considered to be starved, so {{FSPreemptionThread}} calls {{identifyContainersToPreempt}},
and that calls {{FSAppAttempt#getStarvedResourceRequests}} to get a list of {{ResourceRequest}}
instances that are enough to address the app's starvation.
> 2. The first {{ResourceRequest}} that {{getStarvedResourceRequests}} sees is enough to
address the app's starvation, so we break out of the loop over {{appSchedulingInfo.getAllResourceRequests()}}
after only one iteration: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L1180.
We return only this one {{ResourceRequest}} back to the {{identifyContainersToPreempt}} method.
> 3. It turns out that this particular {{ResourceRequest}} happens to have a value for
{{getResourceName}} that identifies a specific node in the cluster. This causes preemption
to only consider containers on that node, and not the rest of the cluster.
> [~kasha], does that make sense? I'm happy to submit a patch if I'm understanding the
problem correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message