hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
Date Thu, 16 Nov 2017 02:25:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254647#comment-16254647

Aaron Fabbri commented on HADOOP-14475:

There is a learning curve for this Metrics2 stuff.  I have some basic questions that are important,
and then a couple of minor nits on the patch.

*Basic Questions*:
{{DefaultMetricsSystem}} lifecycle API is confusing to me.  It appears to have a singleton,
but we are calling {{stop()}} on it from {{S3AFileSystem#close()}}.  What if other stuff in
the JVM is still using the default metrics system when we close the S3A FS?  We also call
{{shutdown()}} on the {{DefaultMetricsSystem}}, but that appears to use refcounts internally
to only really shutdown when the number of  {{shutdown()}} calls reaches the number of {{init()}}
calls.  {{stop()}} does not do this.

The {{MetricsSystemImpl}} interface exposes {{init()}} which calls {{start()}} internally,
but also exposes {{start()}} publicly.  Humm.

Anyways.. I'm guessing we shouldn't be calling {{stop()}} from S3A's {{close()}}.  Seems like
the metrics' {{shutdown()}} calls {{stop()}} when refcount goes to zero?

*Patch feedback*:
+      // Init Metrics systemgti
+      DefaultMetricsSystem.initialize("s3afilesystem");
Typo in comment?
+      String msName=newMetricsSourceName();
+      if(null!=ms){
+    this.recordName=rName;

I guess checkstyle doesn't enforce assignment spacing, but most of the code puts spaces around

+  @VisibleForTesting
+  public static String newMetricsSourceName() {
+    int number = metricsSourceNameCounter.incrementAndGet();
+    final String baseName = "S3AFileSystemMetrics";
+    return baseName + number;
This name is getting long, with the bucket being appended later.  Maybe remove "Metrics" (should
be obvious from context?) and use "S3AFileSystem"?

> Metrics of S3A don't print out  when enable it in Hadoop metrics property file
> ------------------------------------------------------------------------------
>                 Key: HADOOP-14475
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14475
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.8.0
>         Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 x86_64 x86_64
x86_64 GNU/Linux
>  cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>            Reporter: Yonger
>            Assignee: Yonger
>         Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, HADOOP-14475.005.patch,
HADOOP-14475.006.patch, HADOOP-14475.008.patch, HADOOP-14475.009.patch, HADOOP-14775.007.patch,
failsafe-report-s3a-it.html, failsafe-report-s3a-scale.html, failsafe-report-scale.html, failsafe-report-scale.zip,
s3a-metrics.patch1, stdout.zip
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xxxxxxxxxx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message