hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7057) FSAppAttempt#getResourceUsage doesn't need to consider resources queued for preemption
Date Sat, 26 Aug 2017 01:04:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142512#comment-16142512

Karthik Kambatla commented on YARN-7057:

[~templedf] - good questions.

Reasons why getResourceUsage() needs change:
# Subtracting resources queued for preemption before returning current allocation is plain
incorrect, both to users who see this information through UI and other calculations in the
scheduler. It is possible that the application gives up another container, and the current
container queued for preemption is never preempted. Or, more resources are added to the cluster.
Basically, I just feel we shouldn't count our chicken before they hatch.
# Acquiring the lock is an unnecessary overhead all calls to getResourceUsage incur. This
gets called, recursively, so frequently that this can easily add up. Remember the frequency
of calls is the reason we added an if-check in the first place.

I did manually check exhaustively all places getResourceUsage is called. In fact, my first
patch (have not posted) was to have two methods getResourceUsage and getEffectiveResourceUsage.
While working on that version, I carefully considered every call and realized there is only
one place  calling getEffectiveResourceUsage. That is when it dawned on me that I could just
change that one caller. That said, I do appreciate another pair of eyes validate my claim.

> FSAppAttempt#getResourceUsage doesn't need to consider resources queued for preemption
> --------------------------------------------------------------------------------------
>                 Key: YARN-7057
>                 URL: https://issues.apache.org/jira/browse/YARN-7057
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.9.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>         Attachments: YARN-7057.001.patch
> FSAppAttempt#getResourceUsage excludes resources that are currently allocated to the
app but are about to be preempted. This inconsistency shows in the UI and can affect scheduling
of containers. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message