hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7575) When using absolute capacity configuration with no max capacity, scheduler UI NPEs and can't grow queue
Date Tue, 28 Nov 2017 18:54:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269264#comment-16269264
] 

Wangda Tan commented on YARN-7575:
----------------------------------

Thanks [~eepayne],

I just tried to reproduce but failed.

Here's what I do:

1) Setup two queues, both of a/b has capacity: {{\[memory=2048,vcores=8\]}}. Without maximum-capacity
setting. a.user-limit = 100
2) Register single NM with 10GB memory.
3) Run a job requests 100 GB resource. It can use all 10GB memory.
4) Update node resource to 30GB. The job can use all 30GB memory.
5) Update node resource again to 100GB. The job can use all 100GB memory. 

Even though this may not exactly same as your example, I think it should get the same result.
Did I missed anything? 

> When using absolute capacity configuration with no max capacity, scheduler UI NPEs and
can't grow queue
> -------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-7575
>                 URL: https://issues.apache.org/jira/browse/YARN-7575
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>            Reporter: Eric Payne
>         Attachments: YARN-7575-YARN-5881.001.patch
>
>
> I encountered the following while reviewing and testing branch YARN-5881.
> The design document from YARN-5881 says that for max-capacity:
> {quote}
> 3)  For each queue, we require:
>     a) if max-resource not set, it automatically set to parent.max-resource
> {quote}
> When I try leaving blank {{yarn.scheduler.capacity.< queue-path>.maximum-capacity}},
the RMUI scheduler page refuses to render. It looks like it's in {{CapacitySchedulerPage$
LeafQueueInfoBlock}}:
> {noformat}
> 2017-11-28 11:29:16,974 [qtp43473566-220] ERROR webapp.Dispatcher: error handling URI:
/cluster/scheduler
> java.lang.reflect.InvocationTargetException
> ...
>         at org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(CapacitySchedulerPage.java:164)
>         at org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderLeafQueueInfoWithoutParition(CapacitySchedulerPage.java:129)
> {noformat}
> Also... A job will run in the leaf queue with no max capacity set and it will grow to
the max capacity of the cluster, but if I add resources to the node, the job won't grow any
more even though it has pending resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message