hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Helmling (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1956) Export HDFS read and write latency as a metric
Date Tue, 02 Nov 2010 00:38:24 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927237#action_12927237
] 

Gary Helmling commented on HBASE-1956:
--------------------------------------

I believe the reason for incrementing counters in HFile and HLog and then just incrementing
with totals for the polling period in RegionServerMetrics.doUpdates() was to avoid contention
on the synchronized MetricsTimeVaryingRate.inc() call in the critical read and write paths.
 Seems like it would be worth doing some profiling on these in HBASE-3129 to see what the
cost would actually be to call MetricsTimeVaryingRate.inc() per operation instead.

But this definitely makes the FS metrics less useful as a result and squashes the real outliers.

In any case, the bug that omitted fsSyncLatency from the min/max value reset was fixed with
HBASE-3102, and HBASE-3129 addresses improving the min/max values reported, so I think we
can close this issue again.

> Export HDFS read and write latency as a metric
> ----------------------------------------------
>
>                 Key: HBASE-1956
>                 URL: https://issues.apache.org/jira/browse/HBASE-1956
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Andrew Purtell
>            Assignee: Andrew Purtell
>            Priority: Minor
>             Fix For: 0.90.0
>
>         Attachments: HBASE-1956.patch, HBASE-1956.patch
>
>
> HDFS write latency spikes especially are an indicator of general cluster overloading.
We see this where the WAL writer complains about writes taking > 1 second, sometimes >
4, etc.  If for example the average write latency over the monitoring period is exported as
a metric, then this can feed into alerting for or automatic provisioning of additional cluster
hardware. While we're at it, export read side metrics as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message