hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6808) Allow Schedulers to return OPPORTUNISTIC containers when queues go over configured capacity
Date Thu, 13 Jul 2017 18:39:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086186#comment-16086186

Wangda Tan commented on YARN-6808:

[~asuresh], thanks for the detailed explanations. 

I can understand there're two separate targets, but I'm not sure how the 2 targets related
to each other.
1) Use opportunistic container to do lazy preemption in NM. (Is there any umbrella JIRA for
2) Convert guaranteed request to opportunistic request when app's headroom reached.

1) Let's say app1 in an underutilized queue, which want to preempt containers from an over-utilized
queue. Will preemption happens if app1 asks opportunistic container?
2) For target #1, who make the decision of moving guaranteed containers to opportunistic containers.
If it is still decided by central RM, does that mean preemption logics in RM are same as today
except kill operation is decided by NM side? 
3) For overall opportunistic container execution: If OC launch request will be queued by NM,
it may wait a long time before get executed. In this case, do we need to modify AM code to:
a. expect longer delay before think the launch fails. b. asks more resource on different hosts
since there's no guaranteed launch time for OC? 

Comments for target #2. 
- What happens if an app doesn't want to ask opportunistic container when go beyond headroom?
(Such as online services). I think this should be a per-app config (give me OC when I'm go
beyond headroom).
- Existing patch makes static decision, which happens when new resource request added by AM.
Should this be reconsidered when app's headroom changed over time?

Overall, I think this is a big feature and involves lots of components. Including a more detailed
design doc can help contributors understand its scope and workflow.

> Allow Schedulers to return OPPORTUNISTIC containers when queues go over configured capacity
> -------------------------------------------------------------------------------------------
>                 Key: YARN-6808
>                 URL: https://issues.apache.org/jira/browse/YARN-6808
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>         Attachments: YARN-6808.001.patch
> This is based on discussions with [~kasha] and [~kkaranasos].
> Currently, when a Queues goes over capacity, apps on starved queues must wait either
for containers to complete or for them to be pre-empted by the scheduler to get resources.
> This JIRA proposes to allow Schedulers to:
> # Allocate all containers over the configured queue capacity/weight as OPPORTUNISTIC.
> # Auto-promote running OPPORTUNISTIC containers of apps as and when their GUARANTEED
containers complete.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message