Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
Date: Sun, 3 Apr 2016 00:14:19 -0400
Message-ID: 
 <CAFh5nhz6mhUM+fVrzAwLUjy4Po3RD5QYg9Z6sJLVPhMve85Mjw@mail.gmail.com>
Subject: CDH Hbase 1.0.0 Requested row out of range during auto-split
From: Saad Mufti <saad.mufti@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a114707f4962b76052f8cd8af

--001a114707f4962b76052f8cd8af
Content-Type: text/plain; charset=UTF-8

Hi,

We have a large 60 node CDH 5.5.2 Hbase 1.0.0 cluster that take a very
heavy write load. For increased performance, we are using the
BufferedMutator class in hbase-client, although we're using hbase-client
version 1.2.0 because it has a small performance fix to this class.

It seems to be working fine whether the Hbase region servers are undergoing
minor compaction or not, but at some point a lot of them decide to start
auto-splitting. We configure the tables at the beginning with an explicit
set of splits for a total of 1024 regions at the beginning. By now we are
up to 4096 or so regions based on auto-splitting.

We have observed that as soon as auto-splitting starts, the mutations in
the BufferedMutator start throwing exceptions of the following form (after
retries I presume, we are configured for 5 retries and a 3 second RPC
timeout):

BufferedMutator Failed to send mutation
{"totalColumns":1,"row":"0998-f0b05361-5983-46d0-9443-4ca845987aad","families":{"s1":[{"qualifier":"s-10000123-20240198","vlen":28,"tag":[],"timestamp":1459655306487}]},"ttl":72693513}
. org.apache.hadoop.hbase.exceptions.FailedSanityCheckException:
org.apache.hadoop.hbase.exceptions.FailedSanityCheckException: Requested
row out of range for doMiniBatchMutation on HRegion
[region-actual-value-removed],
startKey='0998-UP27f54301-cf7b-11e5-858a-00163ebce99d', getEndKey()='0999',
row='0759-UPbe8df720-c9d5-11e5-947f-00163ee0563a' at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:688)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:639)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1931)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)

Does anyone know the cause for this? I am new to Hbase, in most other
distributed systems I have worked with, the system handles rerouting, so I
was expecting that the Hbase region server would automatically route to the
correct region server/region during splits.

Any help that anyone can provide would be much appreciated. I can provide
more config settings if they turn out to be relevant but there are a ton of
them in Hbase of course and I am not sure which ones would be relevant here.

Thanks in advance.

----

Saad

--001a114707f4962b76052f8cd8af--