Return-Path: X-Original-To: apmail-drill-issues-archive@minotaur.apache.org Delivered-To: apmail-drill-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9C342188C7 for ; Fri, 15 May 2015 01:44:01 +0000 (UTC) Received: (qmail 90455 invoked by uid 500); 15 May 2015 01:44:01 -0000 Delivered-To: apmail-drill-issues-archive@drill.apache.org Received: (qmail 90341 invoked by uid 500); 15 May 2015 01:44:00 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 90100 invoked by uid 99); 15 May 2015 01:44:00 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 May 2015 01:44:00 +0000 Date: Fri, 15 May 2015 01:44:00 +0000 (UTC) From: "Khurram Faraaz (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DRILL-3095) Memory Leak : Failure while closing accountor. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/DRILL-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544754#comment-14544754 ] Khurram Faraaz commented on DRILL-3095: --------------------------------------- The long running query did not Fail, it was Canceled by issuing Ctrl-C. Here is the physical plan for the long running query that was Canceled. {code} 0: jdbc:drill:schema=dfs.tmp> explain plan for select key1, key2 from `twoKeyJsn.json`; +------------+------------+ | text | json | +------------+------------+ | 00-00 Screen 00-01 Project(key1=[$0], key2=[$1]) 00-02 Scan(groupscan=[EasyGroupScan [selectionRoot=/tmp/twoKeyJsn.json, numFiles=1, columns=[`key1`, `key2`], files=[maprfs:/tmp/twoKeyJsn.json]]]) {code} > Memory Leak : Failure while closing accountor. > ---------------------------------------------- > > Key: DRILL-3095 > URL: https://issues.apache.org/jira/browse/DRILL-3095 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow > Affects Versions: 1.0.0 > Environment: f7f6efc525cd833ce1530deae32eb9ccb20b664a > Reporter: Khurram Faraaz > Assignee: Chris Westin > > I am seeing a memory leak when i Cancel a long running query on sqlline. I am re running the query with assertion enabled, will add details after the second run is complete with assertions. > Long running query was, > {code} > select key1, key2 from `twoKeyJsn.json`; > {code} > I did Ctrl-C when the above query was running on sqlline, and then issued the below query that returned correct results. And then I see there is a memory leak message in drillbit.log > {code} > 0: jdbc:drill:schema=dfs.tmp> select count(*) from `twoKeyJsn.json`; > +------------+ > | EXPR$0 | > +------------+ > | 26212355 | > +------------+ > 1 row selected (14.734 seconds) > {code} > Stack trace from drillbit.log > {code} > 2015-05-15 00:59:01,951 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] WARN o.a.drill.exec.ops.SendingAccountor - Interrupted while waiting for send complete. Continuing to wait. > java.lang.InterruptedException: null > at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) ~[na:1.7.0_45] > at java.util.concurrent.Semaphore.acquire(Semaphore.java:472) ~[na:1.7.0_45] > at org.apache.drill.exec.ops.SendingAccountor.waitForSendComplete(SendingAccountor.java:48) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.ops.FragmentContext.waitForSendComplete(FragmentContext.java:436) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.physical.impl.BaseRootExec.close(BaseRootExec.java:112) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.close(ScreenCreator.java:141) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:333) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:278) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > 2015-05-15 00:59:01,952 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2aaabb50-3afd-2906-3f48-eb86a315a1f5:0:0: State change requested from CANCELLATION_REQUESTED --> FAILED for > 2015-05-15 00:59:01,952 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2aaabb50-3afd-2906-3f48-eb86a315a1f5:0:0: State change requested from FAILED --> FAILED for > 2015-05-15 00:59:01,952 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2aaabb50-3afd-2906-3f48-eb86a315a1f5:0:0: State change requested from FAILED --> FINISHED for > 2015-05-15 00:59:01,956 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] ERROR o.a.d.c.exceptions.UserException - SYSTEM ERROR: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are > zone init allocated delta > private 1000000 918080 81920 > shared 9999000000 9999000000 0. > Fragment 0:0 > [Error Id: 90ced8b1-b6db-438f-b193-b7634de31b81 on centos-03.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are > zone init allocated delta > private 1000000 918080 81920 > shared 9999000000 9999000000 0. > Fragment 0:0 > [Error Id: 90ced8b1-b6db-438f-b193-b7634de31b81 on centos-03.qa.lab:31010] > at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522) ~[drill-common-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:315) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:283) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > Caused by: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are > zone init allocated delta > private 1000000 918080 81920 > shared 9999000000 9999000000 0. > at org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.memory.Accountor.close(Accountor.java:386) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.close(TopLevelAllocator.java:325) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:116) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:405) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:394) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:340) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:278) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > ... 4 common frames omitted > 2015-05-15 00:59:01,963 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] INFO o.a.drill.exec.work.foreman.Foreman - State change requested. CANCELLATION_REQUESTED --> FAILED > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are > zone init allocated delta > private 1000000 918080 81920 > shared 9999000000 9999000000 0. > Fragment 0:0 > [Error Id: 90ced8b1-b6db-438f-b193-b7634de31b81 on centos-03.qa.lab:31010] > at org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:481) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.foreman.QueryManager$RootStatusReporter.statusChange(QueryManager.java:461) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.AbstractStatusReporter.fail(AbstractStatusReporter.java:90) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.AbstractStatusReporter.fail(AbstractStatusReporter.java:86) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:319) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:283) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)