hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10175) add per-operation stats to FileSystem.Statistics
Date Tue, 26 Apr 2016 19:13:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258726#comment-15258726
] 

Colin Patrick McCabe commented on HDFS-10175:
---------------------------------------------

bq. I see that, but stream-level counters are essential at least for the tests which verify
forward and lazy seeks. Which means that yes, they do have to go into the 2.8.0 release. What
I can do is set up the scope so that they are package private, then, in the test code, implement
the assertions about metric-derived state into that package.

I guess my hope here is that whatever mechanism we come up with is something that could easily
be integrated into the upcoming 2.8 release.  Since we have talked about requiring our new
metrics to not modify existing stable public interfaces, that seems very reasonable.

One thing that is a bit concerning about metrics2 is that I think people feel that this interface
should be stable (i.e. don't remove or alter things once they're in), which would be a big
constraint on us.  Perhaps we could document that per-fs stats were \@Public \@Evolving rather
than stable?

bq. Regarding the metrics2 instrumentation in HADOOP-13028, I'm aggregating the stream statistics
back into the metrics 2 data. That's something which isn't needed for the hadoop tests, but
which I'm logging in spark test runs, such as (formatted for readability):

Do we have any ideas about how Spark will consume these metrics in the longer term?  Do they
prefer to go through metrics2, for example?  I definitely don't object to putting this kind
of stuff in metrics2, but if we go that route, we have to accept that we'll just get global
(or at best per-fs-type) statistics, rather than per-fs-instance statistics.  Is that acceptable?
 So far, nobody has spoken up strongly in favor of per-fs-instance statistics.

> add per-operation stats to FileSystem.Statistics
> ------------------------------------------------
>
>                 Key: HDFS-10175
>                 URL: https://issues.apache.org/jira/browse/HDFS-10175
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>            Reporter: Ram Venkatesh
>            Assignee: Mingliang Liu
>         Attachments: HDFS-10175.000.patch, HDFS-10175.001.patch, HDFS-10175.002.patch,
HDFS-10175.003.patch, HDFS-10175.004.patch, HDFS-10175.005.patch, HDFS-10175.006.patch, TestStatisticsOverhead.java
>
>
> Currently FileSystem.Statistics exposes the following statistics:
> BytesRead
> BytesWritten
> ReadOps
> LargeReadOps
> WriteOps
> These are in-turn exposed as job counters by MapReduce and other frameworks. There is
logic within DfsClient to map operations to these counters that can be confusing, for instance,
mkdirs counts as a writeOp.
> Proposed enhancement:
> Add a statistic for each DfsClient operation including create, append, createSymlink,
delete, exists, mkdirs, rename and expose them as new properties on the Statistics object.
The operation-specific counters can be used for analyzing the load imposed by a particular
job on HDFS. 
> For example, we can use them to identify jobs that end up creating a large number of
files.
> Once this information is available in the Statistics object, the app frameworks like
MapReduce can expose them as additional counters to be aggregated and recorded as part of
job summary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message