hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Esteban Gutierrez (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13714) Add tracking of the total response queue size
Date Fri, 31 Jul 2015 23:45:05 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14650011#comment-14650011
] 

Esteban Gutierrez commented on HBASE-13714:
-------------------------------------------

This can be useful. For the last few months I've been chasing a memstore/block cache/heap
issue and until I analyzed a heap dump I discovered about 400 calls in-flight to the client
using nearly 300MB of heap (the byte buffer used for the respohnse in some calls was close
to 1MB) The cluster wasn't unstable at all but at certain times the heap utilization was very
close to the maximum and even after flushes there was always an overhead that was not easy
to identify.

> Add tracking of the total response queue size
> ---------------------------------------------
>
>                 Key: HBASE-13714
>                 URL: https://issues.apache.org/jira/browse/HBASE-13714
>             Project: HBase
>          Issue Type: Improvement
>          Components: master, metrics, regionserver, rpc
>    Affects Versions: 2.0.0, 1.0.2, 1.2.0
>            Reporter: Esteban Gutierrez
>
> I noticed this behavior while working on HBASE-13694:
> Once we are done processing a request, we decrement the call queue size on the RPC server.
However, responses can be very large and sometimes sending them can take a long time. Since
we don't keep track the response queue via metrics it is hard to spot when the responses are
using too much resources on the RS. 
> Ideally we should be tracking on the RS how much data we have in-flight in the response
queue via metrics and not just in the logs if the size of the response exceeds a threshold
(e.g hbase.ipc.warn.response.size or hbase.ipc.warn.response.time)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message