hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14862) Metrics for AdlFileSystem
Date Wed, 13 Sep 2017 13:16:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164633#comment-16164633

Steve Loughran commented on HADOOP-14862:

I like the s3a stuff, with some subtle features to replicate

* shared names with HDFS (ideally share with s3a for store ops)
* unsynchronized instrumentation for every stream, values printed in toString(). We've used
for looking at perf issues in downstream code which didn't even need to be recompiled against
Hadoop 2.8, just using LOG.info()
* querying stats and using in tests to count http requests, retries
* serving up as Statistics in FS
* lined up for apps to collect stats of a filesystem (and indeed, an input or output stream,
in future)

Wasb currently lags here, even though its code was the original source of the data

> Metrics for AdlFileSystem
> -------------------------
>                 Key: HADOOP-14862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14862
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/adl
>    Affects Versions: 2.8.0
>            Reporter: John Zhuge
> Add a Metrics2 source {{AdlFileSystemInstrumentation}} for {{AdlFileSystem}}.
> Consider per-thread statistics data if possible. Atomic variables are not totally free
in multi-core arch. Don't think Java can do per-cpu data structure.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message