eagle-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (EAGLE-849) System metric collector python script
Date Thu, 05 Jan 2017 05:55:58 GMT

    [ https://issues.apache.org/jira/browse/EAGLE-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15800426#comment-15800426
] 

ASF GitHub Bot commented on EAGLE-849:
--------------------------------------

Github user haoch commented on a diff in the pull request:

    https://github.com/apache/eagle/pull/763#discussion_r94717251
  
    --- Diff: eagle-external/hadoop_jmx_collector/system_metric_collector.py ---
    @@ -0,0 +1,300 @@
    +# !/usr/bin/python
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +from metric_collector import MetricCollector, Runner
    +import logging, socket, string, os, re, time
    +
    +
    +class SystemMetricCollector(MetricCollector):
    +    METRIC_PREFIX = "system"
    +    METRIC_NAME_EXCLUDE = re.compile(r"[\(|\)]")
    +
    +    def run(self):
    +        self.try_exec_func(
    +            self.collect_cpu_metric,
    +            self.collect_uptime_metric,
    +            self.collect_memory_metric,
    +            self.collect_loadavg_metric,
    +            self.collect_ipmi_cpu_temp,
    +            self.collect_nic_metric,
    +            self.collect_smartdisk_metric,
    +            self.collect_diskstat_metric
    +        )
    +
    +    def try_exec_func(self, *funcs):
    +        for func in funcs:
    +            try:
    +                logging.info("Executing: %s", func.__name__)
    +                func()
    +            except Exception as e:
    +                logging.warn("Failed to execute: %s", func.__name__)
    +                logging.exception(e)
    +
    +    # ====================================
    +    # CPU Usage
    +    # ====================================
    +
    +    def collect_cpu_metric(self):
    +        cpu_metric = self.new_metric()
    +        cpu_info = os.popen('cat /proc/stat').readlines()
    +        demension = ["cpu", "user", "nice", "system", "idle", "wait", "irq", "softirq",
"steal", "guest"]
    +
    +        total_cpu = 0
    +        total_cpu_usage = 0
    +        cpu_stat_pre = None
    +
    +        data_dir = "/tmp/eagle_cpu_stat_previous"
    +        if os.path.exists(data_dir):
    +            fd = open(data_dir, "r")
    +            cpu_stat_pre = fd.read()
    +            fd.close()
    +
    +        for item in cpu_info:
    +            if re.match(r'^cpu\d+', item) is None:
    +                continue
    +
    +            items = re.split("\s+", item.strip())
    +            demens = min(len(demension), len(items))
    +            tuple = dict()
    +            for i in range(1, demens):
    +                tuple[demension[i]] = int(items[i])
    +                cpu_metric['timestamp'] = int(round(time.time() * 1000))
    +                cpu_metric['metric'] = self.METRIC_PREFIX + "." + 'cpu.' + demension[i]
    +                cpu_metric['device'] = items[0]
    +                cpu_metric['value'] = items[i]
    +                self.collect(cpu_metric)
    +
    +            per_cpu_usage = tuple["user"] + tuple["nice"] + tuple["system"] + tuple["wait"]
+ tuple["irq"] + tuple[
    +                "softirq"] + tuple["steal"] + tuple["guest"]
    +            per_cpu_total = tuple["user"] + tuple["nice"] + tuple["system"] + tuple["idle"]
+ tuple["wait"] + tuple[
    +                "irq"] + \
    +                            tuple["softirq"] + tuple["steal"] + tuple["guest"]
    +            total_cpu += per_cpu_total
    +            total_cpu_usage += per_cpu_usage
    --- End diff --
    
    Data Source is:
    ~~~
    [root@sandbox hadoop_jmx_collector]# cat /proc/stat
    cpu  328118 17 134709 9869983 3745 3100 23742 0 0
    cpu0 42170 1 17838 1224218 625 3058 3162 0 0
    cpu1 41435 1 16803 1233688 403 4 2772 0 0
    cpu2 42173 1 17059 1232358 582 6 2706 0 0
    cpu3 40543 1 16920 1234035 471 6 2879 0 0
    cpu4 41876 2 16936 1233251 456 6 2816 0 0
    cpu5 41248 1 16654 1234433 551 5 2664 0 0
    cpu6 41576 1 17046 1233823 335 5 2773 0 0
    cpu7 37094 5 15448 1244174 318 5 3965 0 0
    intr 24785791 145 7 0 0 0 0 0 0 0 0 0 0 112 0 473861 0 0 0 0 138240 350522 0 56 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    ctxt 34107972
    btime 1483582077
    processes 154160
    procs_running 2
    procs_blocked 0
    softirq 26040264 0 8770210 51530 909146 474103 0 2 1651574 154427 14029272
    ~~~
    
    Metric Definition:
    
        CPU Used (per_cpu_usage) = user + nice + system + wait + irq + softirq + steal + guest
        CPU Total (per_cpu_total)    = user + nice + system + idle + wait + irq + softirq
+ steal + guest



> System metric collector python script
> -------------------------------------
>
>                 Key: EAGLE-849
>                 URL: https://issues.apache.org/jira/browse/EAGLE-849
>             Project: Eagle
>          Issue Type: Improvement
>          Components: System Metric Monitor
>    Affects Versions: v0.5.0
>            Reporter: Hao Chen
>            Assignee: Hao Chen
>             Fix For: v0.5.0
>
>
> Refactor System metric collector python script following similar framework as existing
jmx metric collector.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message