Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CAGgZ7qC=KdsfQyQNCbNU0tbqdYLQfgmu5U1f5pjVF0tKZWtfcA@mail.gmail.com>
References: <CAGgZ7qC=KdsfQyQNCbNU0tbqdYLQfgmu5U1f5pjVF0tKZWtfcA@mail.gmail.com>
From: Ted Yu <yuzhihong@gmail.com>
Date: Mon, 24 Oct 2016 07:44:40 -0700
Message-ID: <CALte62zijmb6-sUuG_2ne_PsHG-zGQdPC8aKA10qQV1RJS4efg@mail.gmail.com>
Subject: Re: Spikes in HBase response time
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a114e586e948e2b053f9d6e96
archived-at: Mon, 24 Oct 2016 15:07:07 -0000

--001a114e586e948e2b053f9d6e96
Content-Type: text/plain; charset=UTF-8

Can you take a few jstack traces when this happens (and pastebin the
traces) ?

Do you use Phoenix in this cluster ?

Thanks

On Mon, Oct 24, 2016 at 5:44 AM, Alexander Ilyin <alexander@weborama.com>
wrote:

> Hi,
>
> I'm experiencing some spikes in HBase response time. The problem appeared
> in one moment: before all the requests used to take 4-30ms to complete and
> now some of them (several percent) take as long as 800ms decreasing
> significantly an average processing time.
>
> This happens with the requests to all the tables I have so it doesn't seem
> to be related to some problem in data schema. And it didn't change anyway.
>
> I don't see any problems with the lack of resources: there is enough CPU,
> memory on datanodes.
>
> Automatic splitting is off, no cluster balancing is taking place so it
> shouldn't be the client retries as well.
>
> There are some stacktraces in datanode log but as far as I understand these
> are normal and anyway they had existed before the problem appeared:
> 2016-10-24 05:42:19,186 INFO
> [B.defaultRpcServer.handler=17,queue=2,port=16020]
> shortcircuit.ShortCircuitCache: ShortCircuitCache(0x3a130ae2): could not
> load 1075792477_BP-415105157-10.190.240.206-1468423545420 due to
> InvalidToken exception.
> org.apache.hadoop.security.token.SecretManager$InvalidToken: access
> control
> error while attempting to set up short-circuit access to
> /apps/hbase/data/data/profile/DailyUserScores/
> 84105becaefcf071e43ebc4972d0249f/d/c7653cc98860451cb79e678e9c3f0e85Block
> token with block_token_identifier (expiryDate=1477279957879,
> keyId=2039256459, userId=hbase,
> blockPoolId=BP-415105157-10.190.240.206-1468423545420, blockId=1075792477,
> access modes=[READ]) is expired.
>
> At the moment I'm a bit stuck so any advices on what to check next are
> welcome.
>
> PS. HBase version is 1.1.2
>

--001a114e586e948e2b053f9d6e96--