hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15757) Reverse scan fails with no obvious cause
Date Fri, 06 May 2016 11:41:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273936#comment-15273936
] 

ramkrishna.s.vasudevan commented on HBASE-15757:
------------------------------------------------

Usual questions, how big is the row?  Even after reducing the start and stop row (though you
get result) does it take more time to get back those results?

> Reverse scan fails with no obvious cause
> ----------------------------------------
>
>                 Key: HBASE-15757
>                 URL: https://issues.apache.org/jira/browse/HBASE-15757
>             Project: HBase
>          Issue Type: Bug
>          Components: Client, Scanners
>    Affects Versions: 0.98.12
>         Environment: ubuntu 14.04, amazon cloud; 10 datanodes d2.4xlarge - 16cores, 12x200GB
HDD, 122GB RAM
>            Reporter: Robert Fiser
>
> related issue on stackoverflow: http://stackoverflow.com/questions/37001169/hbase-reverse-scan-error?noredirect=1#comment61558097_37001169
> this works well:
>     scan = new Scan(startRow, stopRow);
> this throws exception sometimes:
>     scan = new Scan(stopRow, startRow);
> 	scan.setReversed(true);
> throwing exception while traffic is at least 100 req/s. there are actually no timeouts,
exception is fired immediately for 1-10% requests
> hbase: 0.98.12-hadoop2;
> hadoop: 2.7.0;
> cluster in AWS, 10 datanodes: d2.4xlarge
> I think it's maybe related with this issue but I'm not using any filters http://apache-hbase.679495.n3.nabble.com/Exception-during-a-reverse-scan-with-filter-td4069721.html
>     	java.lang.RuntimeException: org.apache.hadoop.hbase.DoNotRetryIOException: Failed
after retry of OutOfOrderScannerNextException: was there a rpc timeout?
> 			at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
> 			at com.socialbakers.broker.client.hbase.htable.AbstractHtableListScanner.scanToList(AbstractHtableListScanner.java:30)
> 			at com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:23)
> 			at com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:11)
> 			at com.socialbakers.broker.client.hbase.AbstractHbaseApi.endPointMethod(AbstractHbaseApi.java:40)
> 			at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> 			at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 			at java.lang.reflect.Method.invoke(Method.java:497)
> 			at com.socialbakers.broker.client.Route.invoke(Route.java:241)
> 			at com.socialbakers.broker.client.handler.EndpointHandler.invoke(EndpointHandler.java:173)
> 			at com.socialbakers.broker.client.handler.EndpointHandler.process(EndpointHandler.java:69)
> 			at com.thetransactioncompany.jsonrpc2.server.Dispatcher.process(Dispatcher.java:196)
> 			at com.socialbakers.broker.client.RejectableRunnable.run(RejectableRunnable.java:38)
> 			at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 			at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 			at java.lang.Thread.run(Thread.java:745)
> 			Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException:
was there a rpc timeout?
> 			at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:430)
> 			at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:333)
> 			at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:91)
> 			... 15 more
> 			Caused by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
Expected nextCallSeq: 2 But the nextCallSeq got from client: 1; request=scanner_id: 27700695
number_of_rows: 100 close_scanner: false next_call_seq: 1
> 			at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
> 			at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
> 			at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
> 			at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> 			at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> 			at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> 			at java.lang.Thread.run(Thread.java:745)
> 			at sun.reflect.GeneratedConstructorAccessor16.newInstance(Unknown Source)
> 			at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 			at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> 			at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> 			at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> 			at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:287)
> 			at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:214)
> 			at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
> 			at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115)
> 			at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
> 			at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:375)
> 			... 17 more
> 			Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 2
But the nextCallSeq got from client: 1; request=scanner_id: 27700695 number_of_rows: 100 close_scanner:
false next_call_seq: 1
> 			at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
> 			at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
> 			at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
> 			at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> 			at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> 			at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> 			at java.lang.Thread.run(Thread.java:745)
> 			at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)
> 			at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
> 			at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
> 			at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31392)
> 			at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:173)
> 			... 21 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message