drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kunal Khatua <kkha...@mapr.com>
Subject Re: [Drill 1.10.0] : Memory was leaked by query
Date Tue, 18 Apr 2017 19:43:51 GMT
Could you also share the profiles for the failed queries as well?


Thanks

Kunal

________________________________
From: Padma Penumarthy <ppenumarthy@mapr.com>
Sent: Tuesday, April 18, 2017 7:18:08 AM
To: user@drill.apache.org
Cc: dev@drill.apache.org
Subject: Re: [Drill 1.10.0] : Memory was leaked by query

Seems like you are running into  DRILL-5435<https://issues.apache.org/jira/browse/DRILL-5435>.
Try  turning off async parquet reader and see if that helps.
alter session set `store.parquet.reader.pagereader.async`=false;

Thanks,
Padma


On Apr 18, 2017, at 6:14 AM, Anup Tiwari <anup.tiwari@games24x7.com<mailto:anup.tiwari@games24x7.com>>
wrote:

Hi Team,

Please find following information :

*Cluster configuration :*
Number of Nodes : 5
Cores/Node : 8
RAM : 32

*Variable values :*
planner.width.max_per_node = 5
planner.width.max_per_query = 30
planner.memory.max_query_memory_per_node = 4294967296

I am getting following error on simple select statement which is coming 6
times out of 10 times, let me know if i am missing anything:

*Query :*
select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;

*Error :*

ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)


Fragment 1:24

[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)


Fragment 1:24

[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
   at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)

   at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
   ... 5 common frames omitted
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
--> CANCELLATION_REQUESTED
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.f.FragmentStatusReporter -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
CANCELLATION_REQUESTED
2017-04-18 18:21:54,173 [BitServer-4] WARN
o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO  o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
CANCELLATION_REQUESTED --> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO  o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO  o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FINISHED
2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)


Fragment 1:21

[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)


Fragment 1:21

[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
   at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)

   at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
   at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
   ... 5 common frames omitted


Regards,
*Anup Tiwari*


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message