hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13782) Make MutableRates metrics thread-local write, aggregate-on-read
Date Tue, 08 Nov 2016 19:19:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648515#comment-15648515

Zhe Zhang commented on HADOOP-13782:

Thanks Erik for the update! With HADOOP-13804 change the new class is much cleaner :)

The current concurrency model is still a little complicated. {{snapshot}} has a nested synchronization
on {{globalMetrics}} and {{stat}}, where {{stat}} is a local variable. Maybe we can simplify
the concurrency model by:
# Make {{globalMetrics}} a ConcurrentMap
# Do we want to support multiple threads doing {{snapshot}} at the same time? If not, we should
probably make it a synchronized method so it's easier to maintain and reason about
# Maybe creating a concurrent version of {{SampleStat}}, because that's the only object we
want to protect from concurrent updating (local thread adding, and the snapshotting thread
  private class ConcurrentSampleStat extends SampleStat {
    public synchronized void reset(){
    public synchronized SampleStat add(double x) {
      return super.add(x);
# {{threadLocalMetricsMap}} can be a regular instead of concurrent map?

Also, IIUC, {{snapshot}} is supposed to clear all metrics from the last window. In the v4
patch, if a certain type of metrics appeared in the last window but disappears in the current
window (e.g. thread dies), the entry in {{globalMetrics}} is not cleared.

> Make MutableRates metrics thread-local write, aggregate-on-read
> ---------------------------------------------------------------
>                 Key: HADOOP-13782
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13782
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>            Reporter: Erik Krogen
>            Assignee: Erik Krogen
>         Attachments: HADOOP-13782.000.patch, HADOOP-13782.001.patch, HADOOP-13782.002.patch,
HADOOP-13782.003.patch, HADOOP-13782.004.patch
> Currently the {{MutableRates}} metrics class serializes all writes to metrics it contains
because of its use of {{MetricsRegistry.add()}} (i.e., even two increments of unrelated metrics
contained within the same {{MutableRates}} object will serialize w.r.t. each other). This
class is used by {{RpcDetailedMetrics}}, which may have many hundreds of threads contending
to modify these metrics. Instead we should allow updates to unrelated metrics objects to happen
concurrently. To do so we can let each thread locally collect metrics, and on a {{snapshot}},
aggregate the metrics from all of the threads. 
> I have collected some benchmark performance numbers in HADOOP-13747 (https://issues.apache.org/jira/secure/attachment/12835043/benchmark_results)
which indicate that this can bring significantly higher performance in high contention situations.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message