hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samarth Jain (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-14822) Renewing leases of scanners doesn't work
Date Thu, 24 Dec 2015 01:58:49 GMT

     [ https://issues.apache.org/jira/browse/HBASE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Samarth Jain updated HBASE-14822:
---------------------------------
    Attachment: HBASE-14822_98_nextseq.diff

[~lhofhansl] - I tried out the latest on 0.98, and looks like there are some more issues lurking
with lease renewal. I noticed that on the region server side I was still getting the following
message even though I made sure Phoenix was calling renewLease() for the scanners.

INFO  [RS:0;localhost:55383.leaseChecker] org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener(2633):
Scanner 59 lease expired on region 

After a bit of digging around, it turns out that the lease renewal is actually causing the
regular scan() to fail and vice-versa. This is because renewLease ends up also increasing
the nextCallSeq member variable in the ScannerCallable object. There are checks in place in
the HRegionServer class that causes an OutOfOrderScannerNextException to be thrown because
the nextSeq didn't match. 

See this stacktrace:

org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 2
But the nextCallSeq got from client: 10; request=scanner_id: 56 number_of_rows: 2 close_scanner:
false next_call_seq: 10 renew: false
	at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3277)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31190)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:298)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:216)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
	at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:387)
	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:340)
	at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:57)
	at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:112)
	at 

In this case the number of times renewLease() was called was 8 which also happens to be the
difference between the expected nextCallSeq (2) and the actual nextCallSeq(10). This error
isn't surfaced to the clients though because the HBase client ends up creating a new scanner
altogether behind the scenes. 

One possible simple fix (in the attached patch) would be to not increment the nextCallSeq
when renewing lease. FWIW, after this change, I no longer see the OutOfOrderScannerNextException
and INFO message about scanner lease expiration is also gone.

> Renewing leases of scanners doesn't work
> ----------------------------------------
>
>                 Key: HBASE-14822
>                 URL: https://issues.apache.org/jira/browse/HBASE-14822
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.98.14
>            Reporter: Samarth Jain
>            Assignee: Lars Hofhansl
>             Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.3, 0.98.17, 1.0.4
>
>         Attachments: 14822-0.98-v2.txt, 14822-0.98-v3.txt, 14822-0.98.txt, 14822-v3-0.98.txt,
14822-v4-0.98.txt, 14822-v4.txt, 14822-v5-0.98.txt, 14822-v5-1.3.txt, 14822-v5.txt, 14822.txt,
HBASE-14822_98_nextseq.diff
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message