From dev-return-57177-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Sat Jun 22 06:20:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id B800318076D for ; Sat, 22 Jun 2019 08:20:01 +0200 (CEST) Received: (qmail 79726 invoked by uid 500); 22 Jun 2019 06:20:01 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 79689 invoked by uid 99); 22 Jun 2019 06:20:00 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Jun 2019 06:20:00 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 4C3ACE2CFA for ; Sat, 22 Jun 2019 06:20:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 113D72544A for ; Sat, 22 Jun 2019 06:20:00 +0000 (UTC) Date: Sat, 22 Jun 2019 06:20:00 +0000 (UTC) From: "Lars Hofhansl (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (PHOENIX-5364) Tests fail "Queue Full" only when run in parallel with other tests. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated PHOENIX-5364: ----------------------------------- Description: Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions like this: {code} org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1 at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121) at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182) at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1 at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58) at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315) at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352) ... 11 more {code} This never happens with spooling enabled - it would not use the SizeBoundQueue at all. So something must set spooling to false. In fact I found two more tests that set phoenix.query.server.orderBy.spooling.enabled to false: * OrderByWithServerClientSpoolingDisabledIT, and * OrderByWithServerMemoryLimitIT They need to run in their own cluster. (Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the logs of the jenkins run... Perhaps because it's a direct child of BaseTest...?)) was: Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions like this: {code} org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1 at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121) at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182) at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1 at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58) at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315) at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352) ... 11 more {code} This should never happen. However, I found two more tests that set phoenix.query.server.orderBy.spooling.enabled to false: * OrderByWithServerClientSpoolingDisabledIT, and * OrderByWithServerMemoryLimitIT They need to run in their own cluster. (Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the logs of the jenkins run... Perhaps because it's a direct child of BaseTest...?)) > Tests fail "Queue Full" only when run in parallel with other tests. > ------------------------------------------------------------------- > > Key: PHOENIX-5364 > URL: https://issues.apache.org/jira/browse/PHOENIX-5364 > Project: Phoenix > Issue Type: Test > Reporter: Lars Hofhansl > Assignee: Lars Hofhansl > Priority: Major > Fix For: 4.15.0, 5.1.0 > > Attachments: 5364.txt > > > Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions like this: > {code} > org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1 > at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121) > at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87) > at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357) > at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182) > at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187) > at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251) > at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153) > at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) > Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1 > at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58) > at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315) > at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245) > at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352) > ... 11 more > {code} > This never happens with spooling enabled - it would not use the SizeBoundQueue at all. So something must set spooling to false. > In fact I found two more tests that set phoenix.query.server.orderBy.spooling.enabled to false: > * OrderByWithServerClientSpoolingDisabledIT, and > * OrderByWithServerMemoryLimitIT > They need to run in their own cluster. > (Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the logs of the jenkins run... Perhaps because it's a direct child of BaseTest...?)) -- This message was sent by Atlassian JIRA (v7.6.3#76005)