hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6982) nntop: top­-like tool for name node users
Date Sat, 15 Nov 2014 00:39:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14213177#comment-14213177

Andrew Wang commented on HDFS-6982:

Hi Maysam, took a look at the latest patch, I think we're almost there :) Just minor comments.
Hopefully Jenkins behaves with the next rev too, I agree it looks unrelated or garbled.

DFSConfigKeys / TopConf:
* Need to rename the DFSConfigKeys variable names to reflect new config names
* Seems like I gave bad advice about getInts, since it doesn't have a way of taking a default,
so right now if we try to turn it off, it'll set the default. Reverting to what you had is
cool, though adding a getInts that takes a default would be appreciated.

* Could we add explanatory text to the Precondition checks?

* Rather than injecting it into the conf (kinda brittle), what I had in mind was in FSNamesystem#initAuditLoggers,
we could tack it on the end after adding the ones from the conf. No need for reflection :)
* Related to this, it'd be good to have a unit test that disables nntop and then checks that
the audit logger isn't added and that metrics aren't published. Feel free to add a @VisibleForTesting
getter if it helps.

* Unused import in NameNode

This is just minor stuff though, I'm +1 pending the above review comments.

> nntop: top­-like tool for name node users
> -----------------------------------------
>                 Key: HDFS-6982
>                 URL: https://issues.apache.org/jira/browse/HDFS-6982
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Maysam Yabandeh
>            Assignee: Maysam Yabandeh
>         Attachments: HDFS-6982.patch, HDFS-6982.v2.patch, HDFS-6982.v3.patch, HDFS-6982.v4.patch,
HDFS-6982.v5.patch, HDFS-6982.v6.patch, HDFS-6982.v7.patch, HDFS-6982.v8.patch, nntop-design-v1.pdf
> In this jira we motivate the need for nntop, a tool that, similarly to what top does
in Linux, gives the list of top users of the HDFS name node and gives insight about which
users are sending majority of each traffic type to the name node. This information turns out
to be the most critical when the name node is under pressure and the HDFS admin needs to know
which user is hammering the name node and with what kind of requests. Here we present the
design of nntop which has been in production at Twitter in the past 10 months. nntop proved
to have low cpu overhead (< 2% in a cluster of 4K nodes), low memory footprint (less than
a few MB), and quite efficient for the write path (only two hash lookup for updating a metric).

This message was sent by Atlassian JIRA

View raw message