hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
Date Tue, 28 Nov 2017 04:08:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268073#comment-16268073
] 

Sean Mackrory edited comment on HADOOP-14475 at 11/28/17 4:07 AM:
------------------------------------------------------------------

[~iyonger] I don't believe I've changed anything that would affect whether or not the metrics
source name shows up in the sink's output - I've only changed what the source name would be.
I think the format even when I tested your original patch was pretty much as it is above,
and did not include the source name, only the record name (which happened to be similar).
If you're wanting to aggregate based on the bucket, I would use the bucket field itself. There's
no guarantee that metrics source names would even be consistent among all JVMs for a given
bucket, since they're assigned numbers in the order that they're created - that would only
be true if every JVM had accessed the exact same buckets in the exact same order - the assumption
would break down as soon as a job didn't utilize the entire cluster or a node was down during
a job, etc.


was (Author: mackrorysd):
[~iyonger] I don't believe I've changed anything that would affect whether or not the metrics
source name shows up in the sink's output - I've only changed what the source name would be.
I think the format even when I tested your original patch was pretty much as it is above,
and did not include the source name, only the record name (which happened to be similar).
If you're wanting to aggregate based on the bucket, I would use the bucket field itself. There's
no guarantee that metrics source names would even be consistent among all JVMs for a given
bucket - that would only be true if every JVM had accessed the exact same buckets in the exact
same order - the assumption would break down as soon as a job didn't utilize the entire cluster
or a node was down during a job, etc.

> Metrics of S3A don't print out  when enable it in Hadoop metrics property file
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-14475
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14475
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>         Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 x86_64 x86_64
x86_64 GNU/Linux
>  cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>            Reporter: Yonger
>            Assignee: Yonger
>         Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, HADOOP-14475.005.patch,
HADOOP-14475.006.patch, HADOOP-14475.008.patch, HADOOP-14475.009.patch, HADOOP-14475.010.patch,
HADOOP-14475.011.patch, HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch,
HADOOP-14475.015.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html, failsafe-report-s3a-scale.html,
failsafe-report-scale.html, failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xxxxxxxxxx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message