hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhihai xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3241) Leading space, trailing space and empty sub queue name may cause MetricsException for fair scheduler
Date Tue, 17 Mar 2015 07:09:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364693#comment-14364693
] 

zhihai xu commented on YARN-3241:
---------------------------------

I uploaded a patch YARN-3241.000.patch for review. Without the patch, the test will fail with
MetricsException: Metrics source QueueMetrics,q0=root,q1=A already exists!
{code}
------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Tests run: 87, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.887 sec <<< FAILURE!
- in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
testQueueNameWithTrailingSpace(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
 Time elapsed: 0.036 sec  <<< ERROR!
org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=A already
exists!
	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:143)
	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:120)
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics.forQueue(FSQueueMetrics.java:111)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.<init>(FSQueue.java:63)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.<init>(FSLeafQueue.java:75)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.createQueue(QueueManager.java:185)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getQueue(QueueManager.java:131)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getLeafQueue(QueueManager.java:88)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:699)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:601)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1182)
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueNameWithTrailingSpace(TestFairScheduler.java:4466)
Results :
Tests in error: 
  TestFairScheduler.testQueueNameWithTrailingSpace:4466 ยป Metrics Metrics source...
Tests run: 87, Failures: 0, Errors: 1, Skipped: 0
{code}

> Leading space, trailing space and empty sub queue name may cause MetricsException for
fair scheduler
> ----------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3241
>                 URL: https://issues.apache.org/jira/browse/YARN-3241
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>         Attachments: YARN-3241.000.patch
>
>
> Leading space, trailing space and empty sub queue name may cause MetricsException(Metrics
source XXX already exists! ) when add application to FairScheduler.
> The reason is because QueueMetrics parse the queue name different from the QueueManager.
> QueueMetrics use Q_SPLITTER to parse queue name, it will remove Leading space and trailing
space in the sub queue name, It will also remove empty sub queue name.
> {code}
>   static final Splitter Q_SPLITTER =
>       Splitter.on('.').omitEmptyStrings().trimResults(); 
> {code}
> But QueueManager won't remove Leading space, trailing space and empty sub queue name.
> This will cause out of sync between FSQueue and FSQueueMetrics.
> QueueManager will think two queue names are different so it will try to create a new
queue.
> But FSQueueMetrics will treat these two queue names as same queue which will create "Metrics
source XXX already exists!" MetricsException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message