hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Kinley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8711) Requests count is completely off
Date Wed, 25 Sep 2013 11:38:03 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777370#comment-13777370
] 

James Kinley commented on HBASE-8711:
-------------------------------------


There are a couple of things going on here. The first issue is that the point in time metrics
for a region server are only computed every 15 seconds and therefore requestsPerSecond is
actually an average for the previous interval:

{code}
requestsPerSecond = (numberRequestsSinceLastRun / secondsSinceLastRun)
{code}

The client write buffer also confuses things because the writes are delayed reaching the server,
then the server can take up to 15 seconds to include them in the metrics, and another 15 seconds
to clear them down.

The second issue is that the requestCount counter used to compute requestsPerSecond is incremented
for every region server operation (i.e. including requests for region info and not just for
mutations) and is therefore higher than expected.

I've attached the results of two tests to show whats happening. The same client was used in
both tests to generate 100 req/sec for 60 seconds (6000 writes):

In the first test (RPS-TEST-1.log) the client write buffer was enabled and you can see that
it took 52 seconds before the writes appeared in the metrics, and because they were all sent
at once requestsPerSecond was skewed at 09:07:28 (requestsPerSecond: 602.7862951606452). It
took a further 30 seconds before requestsPerSecond returned to 0. Also, writeRequestsCount
was only incremented once for each flush, and the individual writes were not counted (tempWriteRequestsCount
= 4).

In the second test (RPS-TEST-2.log) the client write buffer was disabled and you can see that
the writes started to appear in the metrics after the next metric computation was run (15
seconds), and requestsPerSecond hovered at ~170 for the duration of the test. The writes were
also counted individually (tempWriteRequestsCount = 6004).

Both tests show an increase of over 12000 requests for 6000 writes.

Computing requestsPerSecond from the requestCount counter makes sense, but the interval should
at least be configurable and be lower by default (maybe 5 seconds).
                
> Requests count is completely off
> --------------------------------
>
>                 Key: HBASE-8711
>                 URL: https://issues.apache.org/jira/browse/HBASE-8711
>             Project: HBase
>          Issue Type: Bug
>          Components: UI
>    Affects Versions: 0.95.1
>            Reporter: Jean-Daniel Cryans
>            Assignee: James Kinley
>             Fix For: 0.96.0
>
>
> I tried 0.95.1 RC1 in standalone, and the requests count in both the master and RS web
UIs are wrong. I haven't dug too much in but it seems too low when I'm sending load, and it
takes >10 seconds to clear up when the cluster becomes completely idle.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message