hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6982) nntop: top­-like tool for name node users
Date Fri, 14 Nov 2014 01:03:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211634#comment-14211634

Andrew Wang commented on HDFS-6982:

I triggered another run manually, haven't looked at latest patch yet, but responding to your

bq. The idea was to increase the periods in exponential manner: 5^0,5^1, 5^2

Sure, if you prefer :) It's all configurable, so I'm easy on this one.

bq. I am inclined towards redundantly specifying the audit logger in the conf. I think it
would also avoid confusion for future readers if we spell out the registered audit loggers.

I see nntop as a new set of metrics, and the fact that it hooks into the audit logger as an
implementation detail that doesn't need to be exposed to clients. Are there any downsides
to automatically registering the TopAuditLogger besides conf visibility? If there isn't already,
we can add a print at NN startup that lists all the audit loggers.

Generally speaking, simplifying the conf is something we try really hard to do. If we do decide
to redundantly specify, we'll need to beef up the docs in hdfs-default.xml a bit to explain

> nntop: top­-like tool for name node users
> -----------------------------------------
>                 Key: HDFS-6982
>                 URL: https://issues.apache.org/jira/browse/HDFS-6982
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Maysam Yabandeh
>            Assignee: Maysam Yabandeh
>         Attachments: HDFS-6982.patch, HDFS-6982.v2.patch, HDFS-6982.v3.patch, HDFS-6982.v4.patch,
HDFS-6982.v5.patch, HDFS-6982.v6.patch, HDFS-6982.v7.patch, nntop-design-v1.pdf
> In this jira we motivate the need for nntop, a tool that, similarly to what top does
in Linux, gives the list of top users of the HDFS name node and gives insight about which
users are sending majority of each traffic type to the name node. This information turns out
to be the most critical when the name node is under pressure and the HDFS admin needs to know
which user is hammering the name node and with what kind of requests. Here we present the
design of nntop which has been in production at Twitter in the past 10 months. nntop proved
to have low cpu overhead (< 2% in a cluster of 4K nodes), low memory footprint (less than
a few MB), and quite efficient for the write path (only two hash lookup for updating a metric).

This message was sent by Atlassian JIRA

View raw message