phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PHOENIX-5364) Tests fail "Queue Full" only when run in parallel with other tests.
Date Sat, 22 Jun 2019 06:20:00 GMT

     [ https://issues.apache.org/jira/browse/PHOENIX-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lars Hofhansl updated PHOENIX-5364:
-----------------------------------
    Description: 
Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions like
this:

{code}
org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException:
N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.:
Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current
size: 77, Number of elements:1
 at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121)
 at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182)
 at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187)
 at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251)
 at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153)
 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold
or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1
 at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58)
 at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315)
 at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352)
 ... 11 more
{code}

This never happens with spooling enabled - it would not use the SizeBoundQueue at all. So
something must set spooling to false.

In fact I found two more tests that set phoenix.query.server.orderBy.spooling.enabled to false:
* OrderByWithServerClientSpoolingDisabledIT, and
* OrderByWithServerMemoryLimitIT

They need to run in their own cluster.

(Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the logs
of the jenkins run... Perhaps because it's a direct child of BaseTest...?))


  was:
Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions like
this:

{code}
org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException:
N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.:
Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current
size: 77, Number of elements:1
 at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121)
 at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182)
 at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187)
 at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251)
 at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153)
 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold
or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1
 at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58)
 at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315)
 at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352)
 ... 11 more
{code}

This should never happen.

However, I found two more tests that set phoenix.query.server.orderBy.spooling.enabled to
false:
* OrderByWithServerClientSpoolingDisabledIT, and
* OrderByWithServerMemoryLimitIT

They need to run in their own cluster.

(Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the logs
of the jenkins run... Perhaps because it's a direct child of BaseTest...?))



> Tests fail "Queue Full" only when run in parallel with other tests.
> -------------------------------------------------------------------
>
>                 Key: PHOENIX-5364
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5364
>             Project: Phoenix
>          Issue Type: Test
>            Reporter: Lars Hofhansl
>            Assignee: Lars Hofhansl
>            Priority: Major
>             Fix For: 4.15.0, 5.1.0
>
>         Attachments: 5364.txt
>
>
> Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions
like this:
> {code}
> org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException:
N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.:
Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current
size: 77, Number of elements:1
>  at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121)
>  at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87)
>  at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357)
>  at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182)
>  at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187)
>  at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251)
>  at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153)
>  at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
>  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
>  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
>  at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
>  at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold
or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1
>  at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58)
>  at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315)
>  at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245)
>  at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352)
>  ... 11 more
> {code}
> This never happens with spooling enabled - it would not use the SizeBoundQueue at all.
So something must set spooling to false.
> In fact I found two more tests that set phoenix.query.server.orderBy.spooling.enabled
to false:
> * OrderByWithServerClientSpoolingDisabledIT, and
> * OrderByWithServerMemoryLimitIT
> They need to run in their own cluster.
> (Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the
logs of the jenkins run... Perhaps because it's a direct child of BaseTest...?))



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message