hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5342) Improve non-exclusive node partition resource allocation in Capacity Scheduler
Date Fri, 15 Jul 2016 07:26:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378954#comment-15378954
] 

Naganarasimha G R commented on YARN-5342:
-----------------------------------------

Hi [~sunilg] & [~wangda],
   Few thoughts on the approach mentioned by both of you 
{quote}
if (Resources.greaterThan(rc,
    application.getCSContext().getClusterResource(),
    application.getCSContext().getClusterResourceUsage()
        .getPending(node.getPartition()), Resources.none()) || StringUtils
    .equals(node.getPartition(), RMNodeLabelsManager.NO_LABEL)) {
  application.resetMissedNonPartitionedRequestSchedulingOpportunity(
      priority);
}
{quote}
Issue i see is we are trying to reverting the counter at application level but trying to evaluate
the pending resources of current node partition. What if multiple non exclusive partitions
are there?
Similar things apply for the approaches captured by Sunil too.
I beleive idea here is to delay the allocation in non exclusive mode till some time so that
preemption doesnt kick in. so would it be a good idea to depend on partition of the node ?


> Improve non-exclusive node partition resource allocation in Capacity Scheduler
> ------------------------------------------------------------------------------
>
>                 Key: YARN-5342
>                 URL: https://issues.apache.org/jira/browse/YARN-5342
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Sunil G
>         Attachments: YARN-5342.1.patch
>
>
> In the previous implementation, one non-exclusive container allocation is possible when
the missed-opportunity >= #cluster-nodes. And missed-opportunity will be reset when container
allocated to any node.
> This will slow down the frequency of container allocation on non-exclusive node partition:
*When a non-exclusive partition=x has idle resource, we can only allocate one container for
this app in every X=nodemanagers.heartbeat-interval secs for the whole cluster.*
> In this JIRA, I propose a fix to reset missed-opporunity only if we have >0 pending
resource for the non-exclusive partition OR we get allocation from the default partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message