hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7630) hadoop-metrics2.properties should have a property *.period set to a default value foe metrics
Date Sun, 18 Sep 2011 21:33:09 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107541#comment-13107541
] 

Eric Yang commented on HADOOP-7630:
-----------------------------------

Luke, It used to be 5 seconds blurb rate (Simon Plugin) and 60 seconds reporting rate (Simon
Aggregator).  Simon Aggregator calculates the wavelet transformation of the output samples.
 Unfortunately, Simon plugin udp datagram does not scale well for having large emission sources.
 Instead of losing blurb (udp datagram), I changed the blurb rate to 60 seconds around August
2008 on Y! clusters.  The Simon plugin is only doing add and average of samples, it does not
have discrete wavelet transformation algorithm built-in.  Consequently, It is better to have
the publish/subscribe rate to be the same as the blurb emission rate so the result is more
accurate.  More averaging computed at the MetricsSource, the less precision is retained at
the source.  

Are you concerning that the metrics might overflow if the publish rate is at 60 seconds? 
In 2008, I had also audited all metrics are using double to prevent overflow in short fraction
of time.  There are few metrics added, but the new metrics do not look like they would overflow
neither.  As a side benefit, by reducing the period, it is less amount of cycle spend in metrics
monitoring, which makes the system more efficient.

> hadoop-metrics2.properties should have a property *.period set to a default value foe
metrics
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7630
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7630
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arpit Gupta
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7630-trunk.patch, HADOOP-7630.patch
>
>
> currently the hadoop-metrics2.properties file does not have a value set for *.period
> This property is useful for metrics to determine when the property will refresh. We should
set it to default of 60

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message