hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ai Deng (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.
Date Thu, 19 Jan 2017 00:35:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829055#comment-15829055
] 

Ai Deng edited comment on HADOOP-13453 at 1/19/17 12:34 AM:
------------------------------------------------------------

[~steve_l] Thank you very much for the explication, that's very helpful. 

I have 2 questions for the moment, for sure there are more to come.

# I see 2 pattern to change the counter value in S3AInstrumentation, have a proper method
like fileCreated() or pass one Statistic to the generic method incrementCounter(), it is for
a reason we keep both? Looks like you suggest to use the second approach.

## I can't find any usage of S3AFileSystem.getStorageStatistics() in the project, what is
the main propose of this statistics? it's for use outside of Hadoop?  I don't need pass an
instance of storageStatistics to S3Guard? In S3AFileSystem, we always increment the both.
{code:java}
  protected void incrementStatistic(Statistic statistic, long count) {
    instrumentation.incrementCounter(statistic, count);
    storageStatistics.incrementCounter(statistic, count);
  }
{code}


was (Author: aid129):
[~steve_l] Thank you very much for the explication, that's very helpful. 

I have 2 questions for the moment, for sure there are more to come.

# I see 2 pattern to change the counter value in S3AInstrumentation, have a proper method
like fileCreated() or pass one Statistic to the generic method incrementCounter(), it is for
a reason we keep both? Looks like you suggest to use the second approach.

# I can't find any usage of S3AFileSystem.getStorageStatistics() in the project, what is the
main propose of this statistics? it's for use outside of Hadoop?  I don't need pass an instance
of storageStatistics to S3Guard? In S3AFileSystem, we always increment the both.
{code:java}
  protected void incrementStatistic(Statistic statistic, long count) {
    instrumentation.incrementCounter(statistic, count);
    storageStatistics.incrementCounter(statistic, count);
  }
{code}

> S3Guard: Instrument new functionality with Hadoop metrics.
> ----------------------------------------------------------
>
>                 Key: HADOOP-13453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13453
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Ai Deng
>
> Provide Hadoop metrics showing operational details of the S3Guard implementation.
> The metrics will be implemented in this ticket:
> ● S3GuardRechecksNthPercentileLatency (MutableQuantiles) ­​ Percentile time spent
> in rechecks attempting to achieve consistency. Repeated for multiple percentile values
> of N.  This metric is an indicator of the additional latency cost of running
S3A with
> S3Guard.
> ● S3GuardRechecksNumOps (MutableQuantiles) ­​ Number of times a consistency
> recheck was required while attempting to achieve consistency.
> ● S3GuardStoreNthPercentileLatency (MutableQuantiles) ­​ Percentile time spent in
> operations against the consistent store, including both write operations during file
system
> mutations and read operations during file system consistency checks. Repeated for
> multiple percentile values of N. This metric is an indicator of latency to the
consistent
> store implementation.
> ● S3GuardConsistencyStoreNumOps (MutableQuantiles) ­​ Number of operations
> against the consistent store, including both write operations during file system mutations
> and read operations during file system consistency checks.
> ● S3GuardConsistencyStoreFailures (MutableCounterLong) ­​ Number of failures
> during operations against the consistent store implementation.
> ● S3GuardConsistencyStoreTimeouts (MutableCounterLong) ­​ Number of timeouts
> during operations against the consistent store implementation.
> ● S3GuardInconsistencies (MutableCounterLong) ­ C​ ount of times S3Guard failed
to
> achieve consistency, even after exhausting all rechecks. A high count may indicate
> unexpected out­of­band modification of the S3 bucket contents, such as by an external
> tool that does not make corresponding updates to the consistent store.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message