hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongwook Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3843) Fair Scheduler should not accept apps with space keys as queue name
Date Mon, 22 Jun 2015 23:30:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596842#comment-14596842
] 

Dongwook Kwon commented on YARN-3843:
-------------------------------------

>From my investigation, QueueMetrics doesn't allow space key string as start or end of
names, it just trims empty strings.

static final Splitter Q_SPLITTER = Splitter.on('.').omitEmptyStrings().trimResults();

https://github.com/apache/hadoop/blob/branch-2.5.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java#L112
https://github.com/apache/hadoop/blob/branch-2.5.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java#L85

So, from FairScheduler, "root.adhoc.birvine ", this queue name with the space at the end of
name, it is treated as different from "root.adhoc.birvine" because it has one more character,
and from QueueMetrics, because names are trimmed, all of sudden, 2 different queue names become
the same that causes the error as "Metrics source QueueMetrics,q0=root,q1=adhoc,q2=birvine
already exists!"


> Fair Scheduler should not accept apps with space keys as queue name
> -------------------------------------------------------------------
>
>                 Key: YARN-3843
>                 URL: https://issues.apache.org/jira/browse/YARN-3843
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.4.0, 2.5.0
>            Reporter: Dongwook Kwon
>            Priority: Minor
>
> As YARN-461, since empty string queue name is not valid, queue name with space keys such
as " " ,"   " should not be accepted either, also not as prefix nor postfix. 
> e.g) "root.test.queuename  ", or "root.test. queuename"
> I have 2 specific cases kill RM with these space keys as part of queue name.
> 1) Without placement policy (hadoop 2.4.0 and above), 
> When a job is submitted with " "(space key) as queue name
> e.g) mapreduce.job.queuename=" "
> 2) With placement policy (hadoop 2.5.0 and above)
>  Once a job is submitted without space key as queue name, and submit another job with
space key.
> e.g) 1st time: mapreduce.job.queuename="root.test.user1" 
> 2nd time: mapreduce.job.queuename="root.test.user1 "
> {code}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.974 sec <<<
FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
> testQueueNameWithSpace(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
 Time elapsed: 0.724 sec  <<< ERROR!
> org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=adhoc,q2=birvine
already exists!
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:135)
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:112)
> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:218)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics.forQueue(FSQueueMetrics.java:96)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.<init>(FSQueue.java:56)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.<init>(FSLeafQueue.java:66)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.createQueue(QueueManager.java:169)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getQueue(QueueManager.java:120)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getLeafQueue(QueueManager.java:88)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:660)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:569)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1127)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueNameWithSpace(TestFairScheduler.java:627)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message