drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacques Nadeau (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (DRILL-4113) memory leak reported while handling query or shutting down
Date Tue, 19 Apr 2016 20:31:25 GMT

     [ https://issues.apache.org/jira/browse/DRILL-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jacques Nadeau resolved DRILL-4113.
-----------------------------------
    Resolution: Cannot Reproduce

> memory leak reported while handling query or shutting down
> ----------------------------------------------------------
>
>                 Key: DRILL-4113
>                 URL: https://issues.apache.org/jira/browse/DRILL-4113
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.3.0, 1.4.0
>            Reporter: Chun Chang
>            Priority: Critical
>
> With impersonation enabled, I've seen two memory leaks. One reported at query time, one
at shutdown.
> At query time:
> {noformat}
> 2015-11-17 19:11:03,595 [29b413b7-958e-c1f3-9d37-c34f96e7bf6a:foreman] INFO  o.a.drill.exec.work.foreman.Foreman
- Query text for query id 29b413b7-958e-c1f3-9d37-c34f96e7bf6a: use `dfs.window_functions`
> 2015-11-17 19:11:03,666 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State change requested AWAITING_ALLOCATION -->
RUNNING
> 2015-11-17 19:11:03,666 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State to report: RUNNING
> 2015-11-17 19:11:03,669 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State change requested RUNNING --> FAILED
> 2015-11-17 19:11:03,669 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State change requested FAILED --> FAILED
> 2015-11-17 19:11:03,669 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State change requested FAILED --> FINISHED
> 2015-11-17 19:11:03,674 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor
- SYSTEM ERROR: IllegalStateException: Failure while closing accountor.  Expected private
and shared pools to be set to initial values.  However, one or more were not.  Stats are
>         zone    init    allocated       delta
>         private 1000000 738112  261888
>         shared  9999000000      9999261888      -261888.
> Fragment 0:0
> [Error Id: 6df67be9-69d4-4a3b-9eae-43ab2404c6d3 on drillats1.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException:
Failure while closing accountor.  Expected private and shared pools to be set to initial values.
 However, one or more were not.  Stats are
>         zone    init    allocated       delta
>         private 1000000 738112  261888
>         shared  9999000000      9999261888      -261888.
> Fragment 0:0
> [Error Id: 6df67be9-69d4-4a3b-9eae-43ab2404c6d3 on drillats1.qa.lab:31010]
>         at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
>         at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: java.lang.IllegalStateException: Failure while closing accountor.  Expected
private and shared pools to be set to initial values.  However, one or more were not.  Stats
are
>         zone    init    allocated       delta
>         private 1000000 738112  261888
>         shared  9999000000      9999261888      -261888.
>         at org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:199)
~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.memory.AccountorImpl.close(AccountorImpl.java:365) ~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.close(TopLevelAllocator.java:326)
~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:123)
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:437)
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:426)
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:346)
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:179)
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         ... 5 common frames omitted
> 2015-11-17 19:11:03,723 [CONTROL-rpc-event-queue] WARN  o.a.drill.exec.work.foreman.Foreman
- Dropping request to move to COMPLETED state as query is already at FAILED state (which is
terminal).
> 2015-11-17 19:11:03,724 [CONTROL-rpc-event-queue] WARN  o.a.d.e.w.b.ControlMessageHandler
- Dropping request to cancel fragment. 29b413b7-edbc-9722-120d-66ab3611f250:0:0 does not exist.
> {noformat}
> At shutdown:
> {noformat}
> Tue Nov 17 22:01:14 PST 2015 Terminating drillbit pid 32653
> 2015-11-17 22:01:14,569 [Drillbit-ShutdownHook#0] INFO  o.apache.drill.exec.server.Drillbit
- Received shutdown request.
> 2015-11-17 22:01:15,559 [BitClient-2] INFO  o.a.drill.exec.rpc.data.DataClient - Channel
closed /10.10.100.171:56037 <--> /10.10.100.172:31012.
> 2015-11-17 22:01:15,559 [BitServer-6] INFO  o.a.d.exec.rpc.control.ControlClient - Channel
closed /10.10.100.171:47521 <--> /10.10.100.172:31011.
> 2015-11-17 22:01:15,574 [BitClient-4] INFO  o.a.drill.exec.rpc.data.DataClient - Channel
closed /10.10.100.171:59422 <--> /10.10.100.174:31012.
> 2015-11-17 22:01:15,575 [BitServer-2] INFO  o.a.d.exec.rpc.control.ControlClient - Channel
closed /10.10.100.171:55874 <--> /10.10.100.174:31011.
> 2015-11-17 22:01:15,582 [BitClient-3] INFO  o.a.drill.exec.rpc.data.DataClient - Channel
closed /10.10.100.171:49728 <--> /10.10.100.173:31012.
> 2015-11-17 22:01:15,582 [BitServer-5] INFO  o.a.d.exec.rpc.control.ControlClient - Channel
closed /10.10.100.171:43599 <--> /10.10.100.173:31011.
> 2015-11-17 22:01:15,603 [BitClient-1] INFO  o.a.drill.exec.rpc.data.DataClient - Channel
closed /10.10.100.171:56089 <--> /10.10.100.171:31012.
> 2015-11-17 22:01:15,603 [BitServer-3] INFO  o.a.d.exec.rpc.control.ControlClient - Channel
closed /10.10.100.171:52294 <--> /10.10.100.171:31011.
> 2015-11-17 22:01:16,610 [pool-1274-thread-1] INFO  o.a.drill.exec.rpc.user.UserServer
- closed eventLoopGroup io.netty.channel.epoll.EpollEventLoopGroup@44d08482 in 1005 ms
> 2015-11-17 22:01:16,610 [pool-1274-thread-2] INFO  o.a.drill.exec.rpc.data.DataServer
- closed eventLoopGroup io.netty.channel.epoll.EpollEventLoopGroup@1a10cc3c in 1007 ms
> 2015-11-17 22:01:16,610 [pool-1274-thread-1] INFO  o.a.drill.exec.service.ServiceEngine
- closed userServer in 1009 ms
> 2015-11-17 22:01:16,610 [pool-1274-thread-2] INFO  o.a.drill.exec.service.ServiceEngine
- closed dataPool in 1009 ms
> 2015-11-17 22:01:16,629 [Drillbit-ShutdownHook#0] WARN  o.apache.drill.exec.server.Drillbit
- Failure on close(): {}
> java.lang.RuntimeException: Exception while closing
>         at org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:127)
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:44) ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:254) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:297)
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> Caused by: java.lang.IllegalStateException: Failure while closing accountor.  Expected
private and shared pools to be set to initial values.  However, one or more were not.  Stats
are
>         zone    init    allocated       delta
>         private 0       0       0
>         shared  34359738368     34358000256     1738112.
>         at org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:199)
~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.memory.AccountorImpl.close(AccountorImpl.java:365) ~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:179)
~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         at org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44)
~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
>         ... 4 common frames omitted
> 2015-11-17 22:01:16,629 [Drillbit-ShutdownHook#0] INFO  o.apache.drill.exec.server.Drillbit
- Shutdown completed (2059 ms).
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message