drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vlad Rozov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-6252) Foreman node is going down when the non foreman node is stopped
Date Fri, 16 Mar 2018 00:02:00 GMT

     [ https://issues.apache.org/jira/browse/DRILL-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vlad Rozov updated DRILL-6252:
------------------------------
    Description: 
Two drillbits are running. I'm running a join query over parquet and tried to stop the non-foreman
node using drillbit.sh stop. The query fails with *"Error: DATA_READ ERROR: Exception occurred
while reading from disk".* The non-foreman node goes down. The foreman node also goes down.
When I looked at the drillbit.log of both foreman and non-foreman I found that there is memory
leak  "Memory was leaked by query. Memory leaked: (2097152)\nAllocator(op:2:0:0:HashPartitionSender)
1000000/6291456/6832128/10000000000 (res/actual/peak/limit)\n". Following are the stack traces
for memory leaks 

{noformat} 
[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Memory
was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)

 

 

Fragment 2:1 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

        at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]

        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)
{noformat} 

 

Ping me for the logs and more information.

 

  was:
Two drillbits are running. I'm running a join query over parquet and tried to stop the non-foreman
node using drillbit.sh stop. The query fails with *"Error: DATA_READ ERROR: Exception occurred
while reading from disk".* The non-foreman node goes down. The foreman node also goes down.
When I looked at the drillbit.log of both foreman and non-foreman I found that there is memory
leak  "Memory was leaked by query. Memory leaked: (2097152)\nAllocator(op:2:0:0:HashPartitionSender)
1000000/6291456/6832128/10000000000 (res/actual/peak/limit)\n". Following are the stack traces
for memory leaks 

 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Memory
was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)

 

 

Fragment 2:1 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

        at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]

        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)

 

Ping me for the logs and more information.

 


> Foreman node is going down when the non foreman node is stopped
> ---------------------------------------------------------------
>
>                 Key: DRILL-6252
>                 URL: https://issues.apache.org/jira/browse/DRILL-6252
>             Project: Apache Drill
>          Issue Type: Bug
>            Reporter: Venkata Jyothsna Donapati
>            Assignee: Vlad Rozov
>            Priority: Critical
>             Fix For: 1.14.0
>
>
> Two drillbits are running. I'm running a join query over parquet and tried to stop the
non-foreman node using drillbit.sh stop. The query fails with *"Error: DATA_READ ERROR: Exception
occurred while reading from disk".* The non-foreman node goes down. The foreman node also
goes down. When I looked at the drillbit.log of both foreman and non-foreman I found that
there is memory leak  "Memory was leaked by query. Memory leaked: (2097152)\nAllocator(op:2:0:0:HashPartitionSender)
1000000/6291456/6832128/10000000000 (res/actual/peak/limit)\n". Following are the stack traces
for memory leaks 
> {noformat} 
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException:
Memory was leaked by query. Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)
>  
>  
> Fragment 2:1 
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
>         at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
>         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked:
(3145728)
> Allocator(op:2:1:0:HashPartitionSender) 1000000/6291456/6291456/10000000000 (res/actual/peak/limit)
> {noformat} 
>  
> Ping me for the logs and more information.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message