incubator-ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom Beerbower (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-2762) Additional YARN metrics required in service status call
Date Tue, 30 Jul 2013 14:33:48 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723906#comment-13723906
] 

Tom Beerbower commented on AMBARI-2762:
---------------------------------------

More Info...

A metric name is mapped to an internal key that is used to find the metric from the backing
source (JMX, Ganglia, ...).  These mappings are maintained in several JSON files.  For example,
the file jmx_properties.json contains an entry like ...

{code}
    "metrics/dfs/FSNamesystem/CapacityTotal":{
        "metric" : "Hadoop:service=NameNode,name=FSNamesystemMetrics.CapacityTotal",
        "pointInTime" : true,
        "temporal" : false
      }
{code}

A request for the metric named 'metrics/dfs/FSNamesystem/CapacityTotal' will result in a request
for JMX data keyed by 'Hadoop:service=NameNode,name=FSNamesystemMetrics.CapacityTotal'.

This mapping of a static metric name to a static key does not work when we allow things like
queue names and hierarchies as part of the metric name.  For example, from JMX we can get
YARN queue metrics like ...

{code}
{
    "name" : "Hadoop:service=ResourceManager,name=QueueMetrics,q0=root,q1=default",
    "modelerType" : "QueueMetrics,q0=root,q1=default",
    "tag.Queue" : "root.default",
    "tag.Context" : "yarn",
    "tag.Hostname" : "ip-10-147-212-220.ec2.internal",
    "running_0" : 0,
    "running_60" : 0,
    "running_300" : 0,
    "running_1440" : 0,
    "AppsSubmitted" : 47,
    "AppsRunning" : 0,
    "AppsPending" : 0,
    "AppsCompleted" : 2,
...
  }
{code}

These metrics are for the queue named "default" under the parent "root", or "/root/default".
In order to support this type of mapping we need to be able to allow template metric names
with argument substitution.  Something like ...

{code}
metrics/yarn/Queue/$1/AppsRunning
{code}

Notice the argument '$1' which can be replaced with the queue name.  To get the queue name
for the substitution we can allow for regular expression matches in the key.  For example
...

{code}
Hadoop:service=ResourceManager,name=QueueMetrics,(.+).AppsRunning
{code}

The JMX metric keyed by 'Hadoop:service=ResourceManager,name=QueueMetrics,q0=root,q1=default.AppsRunning'
would be a regular expression match with 'q0=root,q1=default' being the queue name.

This works but makes for strange metric names like 'metrics/yarn/Queue/q0=root,q1=default/AppsRunning'.
 Ideally we would like to process the queue name so that the metric name would look like 'metrics/yarn/Queue/root/default/AppsRunning'.

To accomplish this, we could add optional methods to the metric name arguments to allow for
some level of processing at substitution time.  For now, we will only support replaceAll()
but we can add more if required in the future.  Keep in mind this is largely an internal feature
as the typical user will not be adding metrics through the default metric providers.

So, an entry in the jmx_properties_2.json file could look like ...

{code}
    "metrics/yarn/Queue$1.replaceAll(\",q(\\d+)=\",\"/\")/AppsRunning":{
        "metric" : "Hadoop:service=ResourceManager,name=QueueMetrics(.+).AppsRunning",
        "pointInTime" : true,
        "temporal" : false
      }
{code}

Notice the replaceAll() method attached to the $1 argument.  The application of this replaceAll
should result in transforming the value of the argument from 'q0=root,q1=default' to 'root/default'
in the above example.  

Also note that this changes the existing behavior in that we have been dropping the queue
name for 'q0=root'.  For example, the JMX metric keyed by ...

{code}
Hadoop:service=ResourceManager,name=QueueMetrics,q0=root
{code}

... currently maps to the metric named ...

{code}
metrics/yarn/Queue/AggregateContainersAllocated
{code}

With the above changes it would instead map to the metric named ...

{code}
metrics/yarn/Queue/root/AggregateContainersAllocated
{code}







                
> Additional YARN metrics required in service status call
> -------------------------------------------------------
>
>                 Key: AMBARI-2762
>                 URL: https://issues.apache.org/jira/browse/AMBARI-2762
>             Project: Ambari
>          Issue Type: Task
>            Reporter: Tom Beerbower
>            Assignee: Tom Beerbower
>         Attachments: AMBARI-2762.patch
>
>
> When client calls /api/v1/clusters/vm/services?fields=components/host_components/metrics/yarn/Queue,
we need following additional information:
>     Queues defined (hierarchy also if possible)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message