Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 20 Jul 2017 00:27:01 +0000 (UTC)
From: "Naganarasimha G R (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13064956.1492569169000.287551.1500510421531@Atlassian.JIRA>
In-Reply-To: <JIRA.13064956.1492569169000@Atlassian.JIRA>
References: <JIRA.13064956.1492569169000@Atlassian.JIRA> <JIRA.13064956.1492569169808@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-6492) Generate queue metrics for each
 partition
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 20 Jul 2017 00:27:08 -0000


    [ https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094003#comment-16094003 ] 

Naganarasimha G R commented on YARN-6492:
-----------------------------------------

Thanks [~manirajv06@gmail.com] for the patch, and sorry for the delay in responding.
I had following high level comments on the approach and scenarios covered:
# User metrics needs to be updated under the partition Queue metric. As we need to capture for each queue under a partition how much resource has been utlized by a user.
# Sorry to retract on the approach, when going through the code i was fealing the approach of the Queue structure is little different from how its displayed in the ui and processed during scheduling as well. Currently we are trying to capture within a queue whats the partition queue's queue metrics. IMO it should be for each partition we should have the queue hierarchy i.e the existing QueueMetrics with user metrics. This would give a better structure
earlier planned approach
{code}
	QueueMetric
		metrics
		Usermetrics
		Partition Metrics
			UserMetrics
		childQueueMetrics
{code}

IMO it should be some thing like 
{code}
	Partitions
		PartitionMetric
			QueueMetric
				metrics
				Usermetrics
				childQueueMetrics
		PartitionMetric
			QueueMetric
			...
	Partitions			
	QueueMetric  //existing for default metrics to maintain compatability
				metrics
				Usermetrics
{code}
But at the same time we should not break the compatability. Thoughts?
May be you can attach the existing structure for a simple queue hierarchy for others to visualize.

Thoughts [~sunilg] , [~jlowe] ?

Implementation
# Ensure the metrics are updated up the tree hierarchy from the leaf till the root on every update on partition metric. I think its taken care but just add test and ensure in manual testing too
# QueueMetrics.java : New overloaded constructor can make a call to existing constructor internally with required arguments instead of both methods doing the update.

Would go through the patch in more detail once the approach gets finalized.


> Generate queue metrics for each partition
> -----------------------------------------
>
>                 Key: YARN-6492
>                 URL: https://issues.apache.org/jira/browse/YARN-6492
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacity scheduler
>            Reporter: Jonathan Hung
>            Assignee: Manikandan R
>         Attachments: YARN-6492.001.patch
>
>
> We are interested in having queue metrics for all partitions. Right now each queue has one QueueMetrics object which captures metrics either in default partition or across all partitions. (After YARN-6467 it will be in default partition)
> But having the partition metrics would be very useful.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org