Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
From: Arun Mishra <arunmishra@me.com>
Content-type: multipart/alternative;
 boundary="Apple-Mail=_E3798EF7-A0BF-4D70-9F20-5BBDAEA5D582"
Subject: Query on OutOfOrderScannerNextException
Message-id: <6C60C13A-F0C9-44BB-9ED3-7817003E5774@me.com>
Date: Sat, 06 Jun 2015 12:54:04 -0700
To: user@hbase.apache.org
MIME-version: 1.0 (Mac OS X Mail 8.2 \(2098\))

--Apple-Mail=_E3798EF7-A0BF-4D70-9F20-5BBDAEA5D582
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Hello,

I have a query on OutOfOrderScannerNextException. I am using hbase =
0.98.6 with 45 nodes.=20

I have a mapreduce job which scan 1 table for last 1 day worth data =
using timerange. It has been running fine for months without any =
failure. But last couple of days it has been failing with below =
exception. I have traced the failure to a single region. This region has =
1 store and 1 hfile of 5+GB. What we realized was that, we were writing =
some bulk data, which used to land on this region. After we stopped =
writing this data, this region has been receiving very few writes per =
day.=20

When mapreduce job runs, it creates a map task for this region and that =
task fails with OutOfOrderScannerNextException. I was able to reproduce =
this error by running a scan command with same start/stop row and =
timerange option. Finally, we split this region to be small enough for =
scan command to work.

My query is if there is any option, apart from increasing the timeout, =
which can solve this use case? I am thinking of a use case where data =
comes in for 3 days a week in bulk and then nothing for next 3 days. =
Kind of creating a data hole in region.
My understanding is that I am hit with this error because I have big =
store files and timerange scan is reading entire file even though it =
contains very few rowkeys for that timerange.

hbase.client.scanner.caching =3D 100
hbase.client.scanner.timeout.period =3D 60s

scan 'dummytable',{ STARTROW=3D>'dummyrowkey-start', =
STOPROW=3D>'dummyrowkey-end', LIMIT=3D>1000, =
TIMERANGE=3D>[1433462400000,1433548800000]}
ROW                                           COLUMN+CELL                =
                                                                         =
                               =20

ERROR: =
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: =
Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; =
request=3Dscanner_id: 33648 number_of_rows: 100 close_scanner: false =
next_call_seq: 0
at =
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java=
:3193)
at =
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.ca=
llBlockingMethod(ClientProtos.java:29587)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at =
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)=

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)


Regards,
Arun=

--Apple-Mail=_E3798EF7-A0BF-4D70-9F20-5BBDAEA5D582--