hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1956) Export HDFS read and write latency as a metric
Date Thu, 31 Dec 2009 04:45:29 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795536#action_12795536
] 

Andrew Purtell commented on HBASE-1956:
---------------------------------------

bq. Do you think we need to do metrics on hfile as well as hlog? 

It depends on what kind of coverage we want these metrics to have.

bq. Would hlog writes be sufficient canary-in-the-mine indicator that hdfs is slow?

Yes, for writes.

bq. I'd be just a little worried about all those System.currentTimeMillis calls.

I don't disagree. 

On HFile read path, only the block read is timed. 

As to the overhead of the call itself, in the OpenJDK source this is a call to gettimeofday
through the JNI. On modern Linux systems, this is a read out of a page mapped into high address
space as a direct memory access, not a syscall. 

On the write side, could only time the writes of an end of block marker?

> Export HDFS read and write latency as a metric
> ----------------------------------------------
>
>                 Key: HBASE-1956
>                 URL: https://issues.apache.org/jira/browse/HBASE-1956
>             Project: Hadoop HBase
>          Issue Type: Improvement
>            Reporter: Andrew Purtell
>            Assignee: Andrew Purtell
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0
>
>         Attachments: HBASE-1956.patch
>
>
> HDFS write latency spikes especially are an indicator of general cluster overloading.
We see this where the WAL writer complains about writes taking > 1 second, sometimes >
4, etc.  If for example the average write latency over the monitoring period is exported as
a metric, then this can feed into alerting for or automatic provisioning of additional cluster
hardware. While we're at it, export read side metrics as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message