hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4512) Provide a knob to turn on over-allocation
Date Wed, 13 Jan 2016 17:36:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096645#comment-15096645

Karthik Kambatla commented on YARN-4512:

bq. do you think is it make sense to let RM make the over-subscription decision?
As mentioned [here|https://issues.apache.org/jira/browse/YARN-1011?focusedCommentId=15072223&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15072223],
here is my reasoning for it being a config on the NM:

Even if we have the knob on the RM, the node still has to support it: monitor the resource
usage on the node and kill the OPPORTUNISTIC containers if need be. On a cluster with NMs
of different versions (say, during a rolling upgrade), the RM will have to keep track of NMs
that support over-subscription. So, we do need some config for the NM anyway. Further, there
could be node-specific conditions - hardware (e.g. GPU), other services running on the node
etc. - that could affect the over-subscription capacity of the node. For instance, it might
be okay to sign up for 90% of the advertised capacity on node A, but only 80% on node B. And,
this ability to soak up extra work could change over time.

bq. And what's the "threshold" means in ResourceThresholds?
Fair question. We plan to use it for both overAllocationThreshold (utilization upto which
the RM allocates OPPORTUNISTIC containers) and preemptionThreshold (utilization beyond which
the node kills enough OPPORTUNISTIC containers). 

My bad, I am updating the design doc to capture these thresholds and the initial scheduling/promotion
policy. Should have it up today or tomorrow. 

> Provide a knob to turn on over-allocation
> -----------------------------------------
>                 Key: YARN-4512
>                 URL: https://issues.apache.org/jira/browse/YARN-4512
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>         Attachments: YARN-4512-YARN-1011.001.patch, yarn-4512-yarn-1011.002.patch, yarn-4512-yarn-1011.003.patch
> We need two configs for overallocation - one to specify the threshold upto which it is
okay to over-allocate, another to specify the threshold after which OPPORTUNISTIC containers
should be preempted.

This message was sent by Atlassian JIRA

View raw message