Return-Path: X-Original-To: apmail-phoenix-dev-archive@minotaur.apache.org Delivered-To: apmail-phoenix-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 42B7C18CEA for ; Thu, 11 Feb 2016 05:55:23 +0000 (UTC) Received: (qmail 36659 invoked by uid 500); 11 Feb 2016 05:55:23 -0000 Delivered-To: apmail-phoenix-dev-archive@phoenix.apache.org Received: (qmail 36600 invoked by uid 500); 11 Feb 2016 05:55:23 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 36589 invoked by uid 99); 11 Feb 2016 05:55:22 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2016 05:55:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 88B73C00E6 for ; Thu, 11 Feb 2016 05:55:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.549 X-Spam-Level: X-Spam-Status: No, score=-3.549 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.329] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id CZUhyitF81VR for ; Thu, 11 Feb 2016 05:55:20 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with SMTP id 1F37B2565F for ; Thu, 11 Feb 2016 05:55:18 +0000 (UTC) Received: (qmail 36578 invoked by uid 99); 11 Feb 2016 05:55:18 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2016 05:55:18 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 250FF2C14F0 for ; Thu, 11 Feb 2016 05:55:18 +0000 (UTC) Date: Thu, 11 Feb 2016 05:55:18 +0000 (UTC) From: "Anoop Sam John (JIRA)" To: dev@phoenix.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-2672) OutOfOrderScannerNextException in SELECT DISTINCT query over tenant connection MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142280#comment-15142280 ] Anoop Sam John commented on PHOENIX-2672: ----------------------------------------- This exception from HBase comes when one scan request within the RS takes too long and client gets timeouts and retry (and old scan request moved some rows forward).. The HBase client will handle such an exception for one time.. It will close the scanner and start from begin. But again we are getting this, means there is some issue which need to be addressed. You are trying to read more data within one scan rpc request. Either reduce the data size to be read or increase the timeouts. > OutOfOrderScannerNextException in SELECT DISTINCT query over tenant connection > ------------------------------------------------------------------------------ > > Key: PHOENIX-2672 > URL: https://issues.apache.org/jira/browse/PHOENIX-2672 > Project: Phoenix > Issue Type: Bug > Affects Versions: 4.6.0 > Reporter: alex kamil > > I'm getting below exception in SELECT DISTINCT query using tenant-specific connection with phoenix 4.6-HBase-1.0-cdh5.5 . > The exception disappears if I either switch to non-tenant connection, or remove DISTINCT from the query. > Caused by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: > Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 2326 number_of_rows: 100 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true > I'm using phoenix 4.6 for cloudera cdh5.5.1 community edition > https://github.com/chiastic-security/phoenix-for-cloudera/tree/4.6-HBase-1.0-cdh5.5 > Below the test case, error log and hbase-site.xml settings: > import java.sql.Connection; > import java.sql.DriverManager; > import java.sql.ResultSet; > import java.sql.SQLException; > import java.sql.Statement; > import java.util.Properties; > public class Test { > public static void main (String [] args){ > Connection conn = null; > String tenant = SYSTEMTENANT; > String url = "my.ip"; > System.out.println("trying to initialize tenant-specific connection to hbaseUrl="+url+" for tenant="+tenant); > Properties connProps = new Properties(); > connProps.setProperty("TenantId", tenant); > String query = "SELECT DISTINCT ROWKEY,VS FROM TABLE1 ORDER BY VS DESC"; > try { > conn = DriverManager.getConnection("jdbc:phoenix:"+url, connProps); > Statement st = conn.createStatement(); > ResultSet resultSet = st.executeQuery(query); > while(resultSet.next()) > { > String rowKey = resultSet.getString(1); > String versionSerial = resultSet.getString(2); > System.out.println("rowkey="+rowKey+", versionserial="+versionSerial); > } > } catch (SQLException e) { > // logger.error(e); > e.printStackTrace(); > } > } > } > Stack trace: > org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout? > at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108) > at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:558) > at org.apache.phoenix.iterate.MergeSortResultIterator.getIterators(MergeSortResultIterator.java:48) > at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:84) > at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:111) > at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64) > at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:771) > at Test.main(Test.java:26) > Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout? > at java.util.concurrent.FutureTask.report(Unknown Source) > at java.util.concurrent.FutureTask.get(Unknown Source) > at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:554) > ... 6 more > Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout? > at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108) > at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:61) > at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:107) > at org.apache.phoenix.iterate.SpoolingResultIterator.(SpoolingResultIterator.java:125) > at org.apache.phoenix.iterate.SpoolingResultIterator.(SpoolingResultIterator.java:83) > at org.apache.phoenix.iterate.SpoolingResultIterator.(SpoolingResultIterator.java:62) > at org.apache.phoenix.iterate.SpoolingResultIterator$SpoolingResultIteratorFactory.newIterator(SpoolingResultIterator.java:78) > at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:109) > at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100) > at java.util.concurrent.FutureTask.run(Unknown Source) > at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: > Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 2326 number_of_rows: 100 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true > at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2177) > at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > at java.lang.Thread.run(Thread.java:744) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:328) > at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:255) > at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:371) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:345) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) > at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64) > hbase-site.xml settings: > > hbase.rpc.timeout > 60000 > > > hbase.client.scanner.caching > 100 > > > phoenix.query.timeoutMs > 60000 > > > "hbase.client.scanner.timeout.period > 60000 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)