hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil Govindan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8720) CapacityScheduler does not enforce yarn.scheduler.capacity.<queue-path>.maximum-allocation-mb/vcores when configured
Date Thu, 13 Sep 2018 11:18:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16613332#comment-16613332
] 

Sunil Govindan commented on YARN-8720:
--------------------------------------

Hi [~tarunparimi]

{{yarn.scheduler.capacity.<queue-path>.maximum-allocation-mb}} and {{yarn.scheduler.capacity.maximum-allocation-mb}}
are both valid. 

And {{Resource getMaximumResourceCapability(String queueName)}} from YarnScheduler, takes
care of falling back to latter configuration.
{code:java}
public Resource getMaximumResourceCapability(String queueName) {
  CSQueue queue = getQueue(queueName);
  if (queue == null) {
    LOG.error("Unknown queue: " + queueName);
    return getMaximumResourceCapability();
  }
  if (!(queue instanceof LeafQueue)) {
    LOG.error("queue " + queueName + " is not an leaf queue");
    return getMaximumResourceCapability();
  }
  return ((LeafQueue)queue).getMaximumAllocation();
}{code}
Hence I think its fine to have your modification. [~cheersyang], could u also please take
a look on this.

> CapacityScheduler does not enforce yarn.scheduler.capacity.<queue-path>.maximum-allocation-mb/vcores
when configured
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-8720
>                 URL: https://issues.apache.org/jira/browse/YARN-8720
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler, capacityscheduler, resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Tarun Parimi
>            Assignee: Tarun Parimi
>            Priority: Major
>             Fix For: 3.2.0
>
>         Attachments: YARN-8720.001.patch, YARN-8720.002.patch
>
>
> The value of yarn.scheduler.capacity.<queue-path>.maximum-allocation-mb/vcores
is not strictly enforced when applications request containers. An InvalidResourceRequestException
is thrown only when the ResourceRequest is greater than the global value of yarn.scheduler.maximum-allocation-mb/vcores
. So for an example configuration such as below,
>  
> {code:java}
> yarn.scheduler.maximum-allocation-mb=4096
> yarn.scheduler.capacity.root.test.maximum-allocation-mb=2048
> {code}
>  
> The below DSShell command runs successfully and asks an AM container of size 4096MB which
is greater than max 2048MB configured in test queue.
> {code:java}
> yarn jar $YARN_HOME/hadoop-yarn-applications-distributedshell.jar -num_containers 1 -jar
$YARN_HOME/hadoop-yarn-applications-distributedshell.jar -shell_command "sleep 60" -container_memory=4096
-master_memory=4096 -queue=test{code}
> Instead it should not launch the application and fail with InvalidResourceRequestException
. The child container however will be requested with size 2048MB as DSShell AppMaster does
the below check before ResourceRequest ask with RM.
> {code:java}
> // A resource ask cannot exceed the max.
> if (containerMemory > maxMem) {
>  LOG.info("Container memory specified above max threshold of cluster."
>  + " Using max value." + ", specified=" + containerMemory + ", max="
>  + maxMem);
>  containerMemory = maxMem;
> }{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message