hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
Date Sun, 11 Dec 2016 19:11:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15740192#comment-15740192

Andrew Purtell commented on HDFS-11194:

It would be an interesting exercise to make a back of the envelope calculation of how much
online storage would be required for the aggregate metrics of say 2000 DataNodes. The patch
on HDFS-10917 adds 6 MutableQuantiles. Would it be just these or all 60 or so metrics in DataNodeMetrics.java
? Assume each metric is a simple long counter for the sake of argument. That would require
8 * 60 * 2000 = ~1 MB of storage. Of course there will be data structure overheads, the quantile
metrics are not single longs, etc. As an operator I can report that maintaining the stability
of NameNodes (2.7.x) with respect to avoiding full GC or the Linux OOM killer is a nontrivial
exercise. Perhaps this feature if implemented could be made optional, but it would be better
to take an approach like Accumulo and implement a separate metrics aggregation service for
the task that can be comfortably run elsewhere than the NN. (See https://accumulo.apache.org/1.7/accumulo_user_manual#_monitor)

> Maintain aggregated peer performance metrics on NameNode
> --------------------------------------------------------
>                 Key: HDFS-11194
>                 URL: https://issues.apache.org/jira/browse/HDFS-11194
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.8.0
>            Reporter: Xiaobing Zhou
>            Assignee: Xiaobing Zhou
> The metrics collected in HDFS-10917 should be reported to and aggregated on NameNode
as part of heart beat messages. This will make is easy to expose it through JMX to users who
are interested in them.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message