hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4512) Provide a knob to turn on over-allocation
Date Sun, 10 Jan 2016 22:43:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15091283#comment-15091283

Karthik Kambatla commented on YARN-4512:

[~elgoiri] - thanks for the quick review. 

bq. Some places (e.g., YarnConfiguration) still use Oversubscription. I prefer that term but
according to the discussion it should be Overallocation everywhere
Should have updated the JIRA. As discussed on YARN-1011, there is a tension between letting
the OPPORTUNISTIC containers run vs preempting them to allow running on a different node.
To address this tension, I feel it would be useful to have a preemption threshold. Given we
have two thresholds, it seemed reasonable to call them allocation and preemption. {{overallocation.allocation}}
seemed somewhat redundant. I am fine with either name. And, we can always change the config
before merging to trunk. 

bq. Should we differentiate physical and virtual memory thresholds?
Since we are not necessarily oversubscribing virtual memory, I don't see the need for virtual
memory thresholds. Any particular reason this would help? 

bq. What would be the use of OverAllocationInfoProto?
This is purely to allow more sophisticated policies on oversubscription instead of just basic
utilization thresholds. For instance, [~leftnoteasy] proposed using history over the last
several time periods. 

bq. Should the RM know about the preemptionThreshold
The preemption happens on the NM. Theoretically, the RM might want to keep track of this and
aggressively make some decisions. No concrete ideas at the moment, so left it out. Any suggestions
on how we could use preemption information on RM? 

Will address rest of the comments in the next iteration. 

> Provide a knob to turn on over-allocation
> -----------------------------------------
>                 Key: YARN-4512
>                 URL: https://issues.apache.org/jira/browse/YARN-4512
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>         Attachments: YARN-4512-YARN-1011.001.patch

This message was sent by Atlassian JIRA

View raw message