eagle-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (EAGLE-928) Refine system metric schema design and fix system metric collector
Date Mon, 27 Feb 2017 10:23:45 GMT

     [ https://issues.apache.org/jira/browse/EAGLE-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hao Chen updated EAGLE-928:
---------------------------
    Description: 
Refine system metric schema design and fix system metric collector

h1. Principle

Any metric (no mater sys metric) should at least have three fields: "metric", "group", "timestamp",
"value", 

"group" is the metric category path separated with slash, for example "FIRST CATEGORY.SECOND
CATEGORY", which will be commonly used for well organizing metric groups in customized metric
dashboard.

h1. Stream Name for System Metric
{code}
<stream>
            <streamId>SYSTEM_METRIC_STREAM</streamId>
            <description>System Metrics Stream including CPU, Network, Disk, etc.</description>
            <columns>
                <column>
                    <name>host</name>
                    <type>string</type>
                </column>
                <column>
                    <name>timestamp</name>
                    <type>long</type>
                </column>
                <column>
                    <name>metric</name>
                    <type>string</type>
                </column>
                <column>
                    <name>group</name>
                    <type>string</type>
                </column>
                <column>
                    <name>site</name>
                    <type>string</type>
                </column>
                <column>
                    <name>device</name>
                    <type>string</type>
                </column>
                <column>
                    <name>value</name>
                    <type>double</type>
                    <defaultValue>0.0</defaultValue>
                </column>
            </columns>
        </stream>
{code}

h1. Stream Schema for System metric
    
       * *metric*: [STRING] metric name string
       * *group*: [STRING] metric group/type
       * *timestamp*: [LONG] metric generation time
       * *site*: [STRING] siteId
       * *host*: [STRING] source host name
       * *device*: [STRING] device name, like cpu

h1. Sample CPU Metric
{code}
{
    "timestamp": 1487918913569, 
    "metric": "system.cpu.usage", 
    "group": "system.cpu",
    "site": "sandbox", 
    "value": 0.058, 
    "host": "sandbox.hortonworks.com", 
    "device": "cpu7"
}
{code}

h1. Sample Network Metrics
{code}
{
    "timestamp": 1487918913569, 
    "metric": "system.nic.transmitdrop", 
    "group": "system.network",
    "site": "sandbox", 
    "value": 7724.0, 
    "host": "sandbox.hortonworks.com", 
    "device": "eth0"
}
{code}

h1. Sample Metric Schema
{code}
{
            "tags": {
                "site": "sandbox",
                "name": "system.memory.nfs_unstable.kb",
                "group": "system.memory"
            },
            "dimensionFields": [
                "host",
                "group",
                "site",
                "device"
            ],
            "metricFields": [
                "value"
            ],
            "granularity": "MINUTE",
            "modifiedTimestamp": 1488190388479
        }
{code}

h1. Stream Persist Topology
{code}
        environment.newApp(config)
                .fromStream("HADOOP_JMX_METRIC_STREAM")
                .saveAsMetric(
                        MetricDescriptor.metricGroupAs((MetricGroupSelector) event -> {
                            if (event.containsKey("component")) {
                                return String.format("hadoop.%s", ((String) event.get("component")).toLowerCase());
                            } else {
                                return "hadoop.metrics";
                            }
                        })
                        .siteByField("site")
                        .namedByField("metric")
                        .eventTimeByField("timestamp")
                        .dimensionFields("host", "component", "site")
                        .granularity(Calendar.MINUTE)
                        .valueField("value"))
                .fromStream("SYSTEM_METRIC_STREAM")
                .saveAsMetric(MetricDescriptor.metricGroupByField("group")
                        .siteByField("site")
                        .namedByField("metric")
                        .eventTimeByField("timestamp")
                        .dimensionFields("host", "group", "site", "device")
                        .granularity(Calendar.MINUTE)
                        .valueField("value")
                )
                .toTopology();
{code}

  was:
Refine system metric schema design and fix system metric collector

h1. Principle

Any metric (no mater sys metric) should at least have three fields: "metric", "group", "timestamp",
"value", 

"group" is the metric category path separated with slash, for example "FIRST CATEGORY.SECOND
CATEGORY", which will be commonly used for well organizing metric groups in customized metric
dashboard.

h1. Stream Name for System Metric
{code}
<stream>
            <streamId>SYSTEM_METRIC_STREAM</streamId>
            <description>System Metrics Stream including CPU, Network, Disk, etc.</description>
            <columns>
                <column>
                    <name>host</name>
                    <type>string</type>
                </column>
                <column>
                    <name>timestamp</name>
                    <type>long</type>
                </column>
                <column>
                    <name>metric</name>
                    <type>string</type>
                </column>
                <column>
                    <name>group</name>
                    <type>string</type>
                </column>
                <column>
                    <name>site</name>
                    <type>string</type>
                </column>
                <column>
                    <name>device</name>
                    <type>string</type>
                </column>
                <column>
                    <name>value</name>
                    <type>double</type>
                    <defaultValue>0.0</defaultValue>
                </column>
            </columns>
        </stream>
{code}
h1. Stream Schema for System metric
    
       * *metric*: [STRING] metric name string
       * *group*: [STRING] metric group/type
       * *timestamp*: [LONG] metric generation time
       * *site*: [STRING] siteId
       * *host*: [STRING] source host name
       * *device*: [STRING] device name, like cpu

h1. Sample CPU Metric
{code}
{
    "timestamp": 1487918913569, 
    "metric": "system.cpu.usage", 
    "group": "system.cpu",
    "site": "sandbox", 
    "value": 0.058, 
    "host": "sandbox.hortonworks.com", 
    "device": "cpu7"
}
{code}

h1. Sample Network Metrics
{code}
{
    "timestamp": 1487918913569, 
    "metric": "system.nic.transmitdrop", 
    "group": "system.network",
    "site": "sandbox", 
    "value": 7724.0, 
    "host": "sandbox.hortonworks.com", 
    "device": "eth0"
}
{code}

h1. Sample Metric Schema
{code}
{
            "tags": {
                "site": "sandbox",
                "name": "system.memory.nfs_unstable.kb",
                "group": "system.memory"
            },
            "dimensionFields": [
                "host",
                "group",
                "site",
                "device"
            ],
            "metricFields": [
                "value"
            ],
            "granularity": "MINUTE",
            "modifiedTimestamp": 1488190388479
        }
{code}


> Refine system metric schema design and fix system metric collector
> ------------------------------------------------------------------
>
>                 Key: EAGLE-928
>                 URL: https://issues.apache.org/jira/browse/EAGLE-928
>             Project: Eagle
>          Issue Type: Bug
>    Affects Versions: v0.5.0
>            Reporter: Hao Chen
>            Assignee: Hao Chen
>             Fix For: v0.5.0
>
>
> Refine system metric schema design and fix system metric collector
> h1. Principle
> Any metric (no mater sys metric) should at least have three fields: "metric", "group",
"timestamp", "value", 
> "group" is the metric category path separated with slash, for example "FIRST CATEGORY.SECOND
CATEGORY", which will be commonly used for well organizing metric groups in customized metric
dashboard.
> h1. Stream Name for System Metric
> {code}
> <stream>
>             <streamId>SYSTEM_METRIC_STREAM</streamId>
>             <description>System Metrics Stream including CPU, Network, Disk, etc.</description>
>             <columns>
>                 <column>
>                     <name>host</name>
>                     <type>string</type>
>                 </column>
>                 <column>
>                     <name>timestamp</name>
>                     <type>long</type>
>                 </column>
>                 <column>
>                     <name>metric</name>
>                     <type>string</type>
>                 </column>
>                 <column>
>                     <name>group</name>
>                     <type>string</type>
>                 </column>
>                 <column>
>                     <name>site</name>
>                     <type>string</type>
>                 </column>
>                 <column>
>                     <name>device</name>
>                     <type>string</type>
>                 </column>
>                 <column>
>                     <name>value</name>
>                     <type>double</type>
>                     <defaultValue>0.0</defaultValue>
>                 </column>
>             </columns>
>         </stream>
> {code}
> h1. Stream Schema for System metric
>     
>        * *metric*: [STRING] metric name string
>        * *group*: [STRING] metric group/type
>        * *timestamp*: [LONG] metric generation time
>        * *site*: [STRING] siteId
>        * *host*: [STRING] source host name
>        * *device*: [STRING] device name, like cpu
> h1. Sample CPU Metric
> {code}
> {
>     "timestamp": 1487918913569, 
>     "metric": "system.cpu.usage", 
>     "group": "system.cpu",
>     "site": "sandbox", 
>     "value": 0.058, 
>     "host": "sandbox.hortonworks.com", 
>     "device": "cpu7"
> }
> {code}
> h1. Sample Network Metrics
> {code}
> {
>     "timestamp": 1487918913569, 
>     "metric": "system.nic.transmitdrop", 
>     "group": "system.network",
>     "site": "sandbox", 
>     "value": 7724.0, 
>     "host": "sandbox.hortonworks.com", 
>     "device": "eth0"
> }
> {code}
> h1. Sample Metric Schema
> {code}
> {
>             "tags": {
>                 "site": "sandbox",
>                 "name": "system.memory.nfs_unstable.kb",
>                 "group": "system.memory"
>             },
>             "dimensionFields": [
>                 "host",
>                 "group",
>                 "site",
>                 "device"
>             ],
>             "metricFields": [
>                 "value"
>             ],
>             "granularity": "MINUTE",
>             "modifiedTimestamp": 1488190388479
>         }
> {code}
> h1. Stream Persist Topology
> {code}
>         environment.newApp(config)
>                 .fromStream("HADOOP_JMX_METRIC_STREAM")
>                 .saveAsMetric(
>                         MetricDescriptor.metricGroupAs((MetricGroupSelector) event ->
{
>                             if (event.containsKey("component")) {
>                                 return String.format("hadoop.%s", ((String) event.get("component")).toLowerCase());
>                             } else {
>                                 return "hadoop.metrics";
>                             }
>                         })
>                         .siteByField("site")
>                         .namedByField("metric")
>                         .eventTimeByField("timestamp")
>                         .dimensionFields("host", "component", "site")
>                         .granularity(Calendar.MINUTE)
>                         .valueField("value"))
>                 .fromStream("SYSTEM_METRIC_STREAM")
>                 .saveAsMetric(MetricDescriptor.metricGroupByField("group")
>                         .siteByField("site")
>                         .namedByField("metric")
>                         .eventTimeByField("timestamp")
>                         .dimensionFields("host", "group", "site", "device")
>                         .granularity(Calendar.MINUTE)
>                         .valueField("value")
>                 )
>                 .toTopology();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message