hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-17213) Add responseTooSlow logging for calls with long totalCallTime
Date Wed, 30 Nov 2016 18:18:58 GMT
Yu Li created HBASE-17213:

             Summary: Add responseTooSlow logging for calls with long totalCallTime
                 Key: HBASE-17213
                 URL: https://issues.apache.org/jira/browse/HBASE-17213
             Project: HBase
          Issue Type: Improvement
            Reporter: Yu Li
            Assignee: Yu Li

Now we only have responseTooSlow logging for long processCallTime, it makes sense not to log
for long queueCallTime since a long processCallTime will cause long queueCallTime for all
succeeding request in queue. However, in our product environment we ever observed the case
that both processCallTime and queueCallTime didn't reach the 10s threshold but totalCallTime
did, but have no idea what the call is against.

Here we propose to add some responseTooSlow logging for totalCallTime of the above case, but
with some logging rate limit to avoid slow processCallTime under heavy load (in which case
call queue might be full and many calls queued thus plenty of long queueCallTime) introduce
too many such logs.

To be explicit, after patch there will be some new log like:
2016-07-27 11:01:12,130 WARN  [PriorityRpcServer.handler=4,queue=0,port=61961] ipc.RpcServer(681):
(responseTooSlow-LongCallTime):  {"processingtimems":5,"call":"Scan(org.apache.hadoop.hbase.protobuf
.generated.ClientProtos$ScanRequest)","client":"","param":"region { type: REGION_NAME
value: \"hbase:meta,,1\" } scan { start_row: \"hbase:namespace,default,99999999999999\" max_versions:
cache_blocks: true small: true reversed: true caching: 1 } number_of_rows: 1 close_scanner:
client_handles_partials: true client_handles_heartbeats: true","starttimems":1469588472124,"queuetimems":1

This message was sent by Atlassian JIRA

View raw message