hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5545) App submit failure on queue with label when default queue partition capacity is zero
Date Fri, 16 Sep 2016 14:14:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496430#comment-15496430
] 

Sunil G commented on YARN-5545:
-------------------------------

Thanks Naga for the points.
Basically we will be giving precedence for any queue specific max-apps config (overriding).
This is existing behavior. So we are only looking for cases where this queue specific config
for max-apps are not present.

bq.if set then there is no need for maxSystemApps * queueCapacities.getAbsoluteCapacity()
as it will never reach
There will not be any defaults for global max-apps config. Hence if its not set specifically
by admin, then we will consider existing way of calculating max-apps for a queue from system
level max-apps w.r.t capacity of queue.

So the code will be reachable if user is not specifying global max-apps config, thus complying
to backward compatibility.

bq.Problem started with maxSystemApps * queueCapacities.getAbsoluteCapacity(), which partition's
absolute capacity needs to be considered when for a given queue is not overriding max applications
and default capacity of the queue is zero. 

I think its a choice from admin side. For scenarios like default label capacity is not configured,
and there are no queue level overriding for max-apps, a possible solution is to configure
global max-apps config. Still if any queue is using its own override, that will be considered.
This can solve the pblm here.
But One case to point out here is that, is there any use case by which customer is expecting
to divide max-apps by capacity (no queue override) and do not want a default global max-apps?.
If so, then we can add few more tuning configs to forcefully enable capacity division of max-apps
in each queue level over default global max-app. Does this make sense?

bq.I feel that enforce strict checking should have been implicit requirement
As mentioned in earlier comment, we can add strict check for max-apps w.r.t to system-wide
max-apps limit. And it can be implicit and we can reject apps if its hit. As pointed out by
Bibin, I do not feel we need a config for that. 

> App submit failure on queue with label when default queue partition capacity is zero
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-5545
>                 URL: https://issues.apache.org/jira/browse/YARN-5545
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Bibin A Chundatt
>            Assignee: Bibin A Chundatt
>         Attachments: YARN-5545.0001.patch, YARN-5545.0002.patch, YARN-5545.0003.patch,
capacity-scheduler.xml
>
>
> Configure capacity scheduler 
> yarn.scheduler.capacity.root.default.capacity=0
> yarn.scheduler.capacity.root.queue1.accessible-node-labels.labelx.capacity=50
> yarn.scheduler.capacity.root.default.accessible-node-labels.labelx.capacity=50
> Submit application as below
> ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha2-SNAPSHOT-tests.jar
sleep -Dmapreduce.job.node-label-expression=labelx -Dmapreduce.job.queuename=default -m 1
-r 1 -mt 10000000 -rt 1
> {noformat}
> 2016-08-21 18:21:31,375 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1471670113386_0001
> java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
application_1471670113386_0001 to YARN : org.apache.hadoop.security.AccessControlException:
Queue root.default already has 0 applications, cannot accept submission of application: application_1471670113386_0001
> 	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:255)
> 	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> 	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1790)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> 	at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:273)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> 	at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:497)
> 	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> 	at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> 	at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:136)
> 	at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:144)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:497)
> 	at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1471670113386_0001
to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has
0 applications, cannot accept submission of application: application_1471670113386_0001
> 	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:286)
> 	at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:296)
> 	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> 	... 25 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message