hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikandan R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6492) Generate queue metrics for each partition
Date Wed, 31 Jan 2018 18:05:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347289#comment-16347289
] 

Manikandan R commented on YARN-6492:
------------------------------------

[~bibinchundatt] Thanks for your comments. Will take care in next patch.

Attaching different runs of JMX o/p's for detailed understanding. These has been captured
in single node setup of 8 GB & 8 vcores and added x,y labels with exclusivity as false.
By default, node has "default" partition.
 # Ran a DS job without any -node_label_expression. Please refer attachment PartitionQueueMetrics_default_partition.txt
 # Mapped node to label 'x'. Ran a DS job with -node_label_expression as "x". Please refer
attachment PartitionQueueMetrics_x_partition.txt
 # Mapped node to label 'y'. Ran a DS job with -node_label_expression as "y". Please refer
attachment PartitionQueueMetrics_y_partition.txt

[~Naganarasimha] [~bibinchundatt]
{quote}I think the assumption that queue metrics is created only during the creation of constructor
is wrong as partition can be added dynamically. hence if the given queuemetrics objects are
not present for a given partition we need to create it.
{quote}
Currently, after RM start, am able to see metrics (for ex, availablememory etc) "default"
partition metrics because of {{CSQueueUtils.updateQueueStatistics(resourceCalculator, clusterResource,
this, labelManager, null)}} in {{AbstractCSQueue#setupQueueConfigs}}. I think we will need
to do this for every node label used in queue configuration with labels. With this, we can
show metrics for all labels after RM start.

For partitions getting added in the middle, Do we need to ensure PartitionQueueMetrics updated
in "replacelabelsonnode" flow? Please share your suggestions.

> Generate queue metrics for each partition
> -----------------------------------------
>
>                 Key: YARN-6492
>                 URL: https://issues.apache.org/jira/browse/YARN-6492
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacity scheduler
>            Reporter: Jonathan Hung
>            Assignee: Manikandan R
>            Priority: Major
>         Attachments: PartitionQueueMetrics_default_partition.txt, PartitionQueueMetrics_x_partition.txt,
PartitionQueueMetrics_y_partition.txt, YARN-6492.001.patch, YARN-6492.002.patch, YARN-6492.003.patch,
partition_metrics.txt
>
>
> We are interested in having queue metrics for all partitions. Right now each queue has
one QueueMetrics object which captures metrics either in default partition or across all partitions.
(After YARN-6467 it will be in default partition)
> But having the partition metrics would be very useful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message