drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khurram Faraaz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-6229) memory leak when output_batch_size is set to 4096
Date Tue, 20 Mar 2018 21:31:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16407091#comment-16407091
] 

Khurram Faraaz commented on DRILL-6229:
---------------------------------------

[~ppenumarthy] we have a test to reproduce the memory leak out of the test framework. I have
attached the test program here, I can share the data file when you debug the problem.

Run the test program with the default setting of output batch size option and we will hit
the memory leak.

Stack trace from the prompt where the concurrency program was executed.
{noformat}
14:02:05.551 [Client-1] INFO o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding
remaining 191167 byte(s) in buffer.
14:02:05.552 [pool-5-thread-2] WARN o.apache.drill.jdbc.DrillConnection - Exception while
closing client.
java.lang.RuntimeException: Exception while closing
at org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.exec.client.DrillClient.close(DrillClient.java:475) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810)
[drill-jdbc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823) [drill-jdbc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36) [drill-jdbc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205) [avatica-1.10.0.jar:1.10.0]
at ConcurrentQuery.executeQuery(ConcurrentQuery.java:84) [concur/:na]
at ConcurrentQuery.selectData(ConcurrentQuery.java:58) [concur/:na]
at ConcurrentQuery.run(ConcurrentQuery.java:32) [concur/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (4194304)
Allocator(ROOT) 0/0/39845888/28631367680 (res/actual/peak/limit)

at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520) ~[drill-memory-base-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
... 13 common frames omitted{noformat}
Query used was :
{noformat}
[root@qa102-45 concur]# cat testSQL.q
select flatten(array_of_ints) from tbl_all_types_jsn_to_parquet{noformat}

> memory leak when output_batch_size is set to 4096
> -------------------------------------------------
>
>                 Key: DRILL-6229
>                 URL: https://issues.apache.org/jira/browse/DRILL-6229
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow
>    Affects Versions: 1.12.0
>            Reporter: Khurram Faraaz
>            Assignee: Padma Penumarthy
>            Priority: Major
>
> When drill.exec.memory.operator.output_batch_size` = 4096 (4KB), we continuously see
in each test run, the memory leak when tests are run from test framework.
> Drill 1.13.0 git.commit.id=766315ea17377199897d685ab801edd38394fe01
> alter system set `drill.exec.memory.operator.output_batch_size` = 4096;
> To run flatten tests from drill test framework
> bin/run_tests -s Functional/flatten_operators -g functional -d -t 500 -n 40
> The below two tests time out and we see the memory leak
> Timeouts:
> /root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/100000rows/filter24.q
> Timeouts:
> /root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/100000rows/filter19.q
> oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection ...
(user client) closed unexpectedly. Drillbit down?
> Stack trace printed in drill test framework log
> {noformat}
> [Error Id: 54a16569-6555-4e92-aa5f-f76a8c11b31f ]
> at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:376)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1003)
> at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:632)
> at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:610)
> at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:553)
> at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1236)
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
> at oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
> at oadd.io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
> at oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
> at oadd.io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
> at oadd.io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:35)
> at oadd.io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:607)
> at oadd.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
> at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
> at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
> at java.lang.Thread.run(Thread.java:748)
> Channel is closed, discarding remaining 908 byte(s) in buffer.
> Exception while closing client.
> java.lang.RuntimeException: Exception while closing
> at oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
> at oadd.org.apache.drill.exec.client.DrillClient.close(DrillClient.java:459)
> at org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810)
> at org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823)
> at org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36)
> at oadd.org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205)
> at org.apache.drill.test.framework.ConnectionPool.close(ConnectionPool.java:102)
> at org.apache.drill.test.framework.TestDriver.runTests(TestDriver.java:562)
> at org.apache.drill.test.framework.TestDriver.main(TestDriver.java:100)
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked:
(4198400)
> Allocator(ROOT) 0/4096/6713344/28631367680 (res/actual/peak/limit)
> at oadd.org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520)
> at oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44)
> ... 8 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message