hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Ilyin <alexan...@weborama.com>
Subject Spikes in HBase response time
Date Mon, 24 Oct 2016 12:44:45 GMT

I'm experiencing some spikes in HBase response time. The problem appeared
in one moment: before all the requests used to take 4-30ms to complete and
now some of them (several percent) take as long as 800ms decreasing
significantly an average processing time.

This happens with the requests to all the tables I have so it doesn't seem
to be related to some problem in data schema. And it didn't change anyway.

I don't see any problems with the lack of resources: there is enough CPU,
memory on datanodes.

Automatic splitting is off, no cluster balancing is taking place so it
shouldn't be the client retries as well.

There are some stacktraces in datanode log but as far as I understand these
are normal and anyway they had existed before the problem appeared:
2016-10-24 05:42:19,186 INFO
shortcircuit.ShortCircuitCache: ShortCircuitCache(0x3a130ae2): could not
load 1075792477_BP-415105157- due to
InvalidToken exception.
org.apache.hadoop.security.token.SecretManager$InvalidToken: access control
error while attempting to set up short-circuit access to
token with block_token_identifier (expiryDate=1477279957879,
keyId=2039256459, userId=hbase,
blockPoolId=BP-415105157-, blockId=1075792477,
access modes=[READ]) is expired.

At the moment I'm a bit stuck so any advices on what to check next are

PS. HBase version is 1.1.2

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message