hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maysam Yabandeh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6982) nntop: top­-like tool for name node users
Date Tue, 02 Sep 2014 23:41:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118971#comment-14118971

Maysam Yabandeh commented on HDFS-6982:

Thanks [~philip]. I agree with you. I actually was planning to skip the audit log tailing
stuff altogether to keep the patch simple. If there was interest in future I can submit a
separate patch for that.

The metric key format is "operation.user". Here is a sample output from the jmx interface:
[myabandeh@smf1-aro-39-sr1(hadoop-tst-nn) ~]$ curl localhost:12333/jmx | grep Hadoop:service=nntop,name=topusers
-B1 -A8
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  }, {
    "name" : "Hadoop:service=nntop,name=topusers",
    "modelerType" : "topusers",
    "tag.Context" : "namenode",
    "tag.ProcessName" : "DummyProcessName",
    "tag.SessionId" : "DummySessionId",
    "tag.Hostname" : "hhh",
    "delete.xxx" : 1,
    "setPermission.ALL" : 0,
    "getfileinfo.ALL" : 3159,

> nntop: top­-like tool for name node users
> -----------------------------------------
>                 Key: HDFS-6982
>                 URL: https://issues.apache.org/jira/browse/HDFS-6982
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Maysam Yabandeh
>         Attachments: nntop-design-v1.pdf
> In this jira we motivate the need for nntop, a tool that, similarly to what top does
in Linux, gives the list of top users of the HDFS name node and gives insight about which
users are sending majority of each traffic type to the name node. This information turns out
to be the most critical when the name node is under pressure and the HDFS admin needs to know
which user is hammering the name node and with what kind of requests. Here we present the
design of nntop which has been in production at Twitter in the past 10 months. nntop proved
to have low cpu overhead (< 2% in a cluster of 4K nodes), low memory footprint (less than
a few MB), and quite efficient for the write path (only two hash lookup for updating a metric).

This message was sent by Atlassian JIRA

View raw message