hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5783) Unit tests to verify the identification of starved applications
Date Mon, 31 Oct 2016 20:39:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623344#comment-15623344

Karthik Kambatla commented on YARN-5783:

bq. If you're assuming a single-threaded context, there's no need to null out the appBeingProcessed
in take().
By nulling out before the blocking call, we are avoiding a match when adding a starved app.

For the scenario outlined, appBeingProcessed is set to app1 only when the preemption thread
is processing it trying to identify nodes that would match. As soon as that processing is
done, appBeingProcessed is reset to null. If the app continues to be starved, the update thread
attempts to queue it. The app is added to FSStarvedApps any time it is not actively processed
by the preemption thread. 

bq. I don't see where tracking the app being processed is needed for the tests.
While implementing the test, I noticed a couple of things - (1) preemption kicks in fast enough
that it does not make much sense to look at the latest snapshot of starved apps, (2) The app
is added multiple times if we don't track appBeingProcessed. 

In general, since the first patch has no tests, adding more tests will lead to finding issues
that need to be fixed. 

Also, this branch is weird in that this could just be one JIRA instead of multiple JIRAs/commits.
The branch is only so the development is in the open and reviews stay small. For the merge,
I would likely want to commit the entire branch as a single commit. So, as long as the final
output is reasonable, I wouldn't worry too much about which JIRA adds what. 

> Unit tests to verify the identification of starved applications
> ---------------------------------------------------------------
>                 Key: YARN-5783
>                 URL: https://issues.apache.org/jira/browse/YARN-5783
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler
>    Affects Versions: 2.8.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>              Labels: oct16-medium
>         Attachments: yarn-5783.YARN-4752.1.patch, yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch
> JIRA to track unit tests to verify the identification of starved applications. An application
should be marked starved only when:
> # Cluster allocation is over the configured threshold for preemption.
> # Preemption is enabled for a queue and any of the following:
> ## The queue is under its minshare for longer than minsharePreemptionTimeout
> ## One of the queue’s applications is under its fairshare for longer than fairsharePreemptionTimeout.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message