hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13782) Make MutableRates metrics thread-local write, aggregate-on-read
Date Wed, 09 Nov 2016 00:21:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649310#comment-15649310
] 

Hudson commented on HADOOP-13782:
---------------------------------

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10793 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10793/])
HADOOP-13782. Make MutableRates metrics thread-local write, (zhz: rev 77c13c385774c51766fe505397fa916754ac08d4)
* (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcDetailedMetrics.java
* (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java
* (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java
* (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java
* (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRates.java


> Make MutableRates metrics thread-local write, aggregate-on-read
> ---------------------------------------------------------------
>
>                 Key: HADOOP-13782
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13782
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>            Reporter: Erik Krogen
>            Assignee: Erik Krogen
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
>         Attachments: HADOOP-13782.000.patch, HADOOP-13782.001.patch, HADOOP-13782.002.patch,
HADOOP-13782.003.patch, HADOOP-13782.004.patch, HADOOP-13782.005.patch, HADOOP-13782.006.patch
>
>
> Currently the {{MutableRates}} metrics class serializes all writes to metrics it contains
because of its use of {{MetricsRegistry.add()}} (i.e., even two increments of unrelated metrics
contained within the same {{MutableRates}} object will serialize w.r.t. each other). This
class is used by {{RpcDetailedMetrics}}, which may have many hundreds of threads contending
to modify these metrics. Instead we should allow updates to unrelated metrics objects to happen
concurrently. To do so we can let each thread locally collect metrics, and on a {{snapshot}},
aggregate the metrics from all of the threads. 
> I have collected some benchmark performance numbers in HADOOP-13747 (https://issues.apache.org/jira/secure/attachment/12835043/benchmark_results)
which indicate that this can bring significantly higher performance in high contention situations.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message