hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
Date Mon, 20 Nov 2017 18:50:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259646#comment-16259646
] 

Steve Loughran commented on HADOOP-14475:
-----------------------------------------

Looking good.



h3. S3AInstrumentation

* S3AInstrumentation: how about making it Closeable and then use IOUtils.closeQuietly() in
S3AFileSystem?
* import layout: all org.apache.hadoop entries should be in their own block. I know a lot
of code doesn't do that, but we shouldn't add to the problem
* S3AInstrumentation.CONTEXT: add a javadoc explaining what this (public) constant does; include
the {{@value}} reference.
* L209. Move comment down a line

h3. index.md

* use `` quotes for things like {{hadoop-metrics2.properties}} in the text
* Add an xref to the metrics itself
* Maybe include a few lines of the outtput of a log to file, to show people what to expect
* Warn that S3A metrics don't include the low-level metrics from inside the AWS SDK, so retry
and throttling events at that level do not get collected.


Collection of the AWS metrics is something it'd be nice to add. It seems possible to get at
them.

h3. ITestS3AMetrics


L41: add enough to the exception message that someone seeing the stack trace on jenkins could
be able to start debugging. e.g 

"No metrics under " + fs + " for " + targetMetricSource,



> Metrics of S3A don't print out  when enable it in Hadoop metrics property file
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-14475
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14475
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.8.0
>         Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 x86_64 x86_64
x86_64 GNU/Linux
>  cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>            Reporter: Yonger
>            Assignee: Yonger
>         Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, HADOOP-14475.005.patch,
HADOOP-14475.006.patch, HADOOP-14475.008.patch, HADOOP-14475.009.patch, HADOOP-14475.010.patch,
HADOOP-14475.011.patch, HADOOP-14475.012.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
failsafe-report-s3a-scale.html, failsafe-report-scale.html, failsafe-report-scale.zip, s3a-metrics.patch1,
stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xxxxxxxxxx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message