hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3881) Writing RM cluster-level metrics
Date Thu, 02 Jul 2015 19:57:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612429#comment-14612429

Zhijie Shen commented on YARN-3881:

IMHO, we need to add an addition API to direct write the cross app metrics (or already aggregated
metrics, if you think of these ones are actually the aggregated data of each individual app,
such as the counters of submitted/pending/running apps) to the backend, in the separate tables,
such as cluster/queue/user tables, and these data don't need to be aggregated any more.

> Writing RM cluster-level metrics
> --------------------------------
>                 Key: YARN-3881
>                 URL: https://issues.apache.org/jira/browse/YARN-3881
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>         Attachments: metrics.json
> RM has a bunch of metrics that we may want to write into the timeline backend to. I attached
the metrics.json that I've crawled via {{http://localhost:8088/jmx?qry=Hadoop:*}}. IMHO, we
need to pay attention to three groups of metrics:
> 1. QueueMetrics
> 2. JvmMetrics
> 3. ClusterMetrics
> The problem is that unlike other metrics belongs to a single application, these ones
belongs to RM or cluster-wide. Therefore, current write path is not going to work for these
metrics because they don't have the associated user/flow/app context info. We need to rethink
of modeling cross-app metrics and the api to handle them.

This message was sent by Atlassian JIRA

View raw message