drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roman (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-5083) RecordIterator can sometimes restart a query on close
Date Wed, 19 Jul 2017 17:40:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093471#comment-16093471
] 

Roman edited comment on DRILL-5083 at 7/19/17 5:39 PM:
-------------------------------------------------------

It seems I got some reproduce of this issue. 
I use Drill from master (35d07c3bd) which includes [DRILL-5420|https://issues.apache.org/jira/browse/DRILL-5420]
and [DRILL-5599|https://issues.apache.org/jira/browse/DRILL-5599] fixes (CANCELLATION_REQUESTED
issues). Here is a list of my properties:

planner.enable_hashjoin = false;
planner.enable_hashagg = false;
planner.enable_mergejoin = true;
planner.memory.max_query_memory_per_node = 1048576;

I ran query (I use tpcds_sf100-query2 on parquet tables) which should fail after ~2 min with
"RESOURCE ERROR: External Sort encountered an error while spilling to disk" and manually cancelled
it after 1min 40 sec. In this case query hangs in CANCELLATION_REQUESTED state until I restart
drillbit. It seems query hangs in code generation state. Here is my jstack example:
{code:xml}
   "26989d8b-2aa4-1a56-f7d3-2b1d7b55d786:frag:10:1" #181 daemon prio=10 os_prio=0 tid=0x00007f1eec0b0800
nid=0x2be0 sleeping[0x00007f1edc650000]
   java.lang.Thread.State: RUNNABLE
	at com.sun.codemodel.JStringLiteral.generate(JStringLiteral.java:61)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:363)
	at com.sun.codemodel.JInvocation.generate(JInvocation.java:185)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JThrow.state(JThrow.java:67)
	at com.sun.codemodel.JFormatter.s(JFormatter.java:386)
	at com.sun.codemodel.JBlock.generateBody(JBlock.java:448)
	at com.sun.codemodel.JBlock.generate(JBlock.java:436)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JConditional.state(JConditional.java:115)
	at com.sun.codemodel.JFormatter.s(JFormatter.java:386)
	at com.sun.codemodel.JBlock.generateBody(JBlock.java:448)
	at com.sun.codemodel.JBlock.generate(JBlock.java:436)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JBlock.state(JBlock.java:464)
	at com.sun.codemodel.JFormatter.s(JFormatter.java:386)
	at com.sun.codemodel.JBlock.generateBody(JBlock.java:448)
	at com.sun.codemodel.JBlock.generate(JBlock.java:436)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JBlock.state(JBlock.java:464)
	at com.sun.codemodel.JFormatter.s(JFormatter.java:386)
	at com.sun.codemodel.JMethod.declare(JMethod.java:460)
	at com.sun.codemodel.JFormatter.d(JFormatter.java:376)
	at com.sun.codemodel.JDefinedClass.declareBody(JDefinedClass.java:815)
	at com.sun.codemodel.JDefinedClass.declare(JDefinedClass.java:788)
	at com.sun.codemodel.JFormatter.d(JFormatter.java:376)
	at com.sun.codemodel.JFormatter.write(JFormatter.java:406)
	at com.sun.codemodel.JPackage.build(JPackage.java:438)
	at com.sun.codemodel.JCodeModel.build(JCodeModel.java:311)
	at com.sun.codemodel.JCodeModel.build(JCodeModel.java:301)
	at org.apache.drill.exec.expr.CodeGenerator.generate(CodeGenerator.java:191)
	at org.apache.drill.exec.compile.CodeCompiler.createInstances(CodeCompiler.java:177)
	at org.apache.drill.exec.compile.CodeCompiler.createInstance(CodeCompiler.java:159)
	at org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:325)
	at org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:319)
	at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.generateNewWorker(MergeJoinBatch.java:382)
	at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:193)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
	at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
	at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
	at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:325)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
	at org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:140)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
	at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144)
	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
	at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
	at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
	at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
	at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x00000000e2398088> (a java.util.concurrent.ThreadPoolExecutor$Worker)

{code}

I got not the same jstack as was described in previous messages, but in this case, I can get
an infinite loop in MergeJoin and I think it relates to this issue.


was (Author: romankulyk):
It seems I got some reproduce of this issue. 
I use Drill from master (35d07c3bd) which includes [DRILL-5420|https://issues.apache.org/jira/browse/DRILL-5420]
and [DRILL-5599|https://issues.apache.org/jira/browse/DRILL-5599] fixes (CANCELLATION_REQUESTED
issues). Here is a list of my properties:

planner.enable_hashjoin = false;
planner.enable_hashagg = false;
planner.enable_mergejoin = true;
planner.memory.max_query_memory_per_node = 1048576;

I ran query which should fail after ~2 min with "RESOURCE ERROR: External Sort encountered
an error while spilling to disk" and manually cancelled it after 1min 40 sec. In this case
query hangs in CANCELLATION_REQUESTED state until I restart drillbit. It seems query hangs
in code generation state. Here is my jstack example:
{code:xml}
   "26989d8b-2aa4-1a56-f7d3-2b1d7b55d786:frag:10:1" #181 daemon prio=10 os_prio=0 tid=0x00007f1eec0b0800
nid=0x2be0 sleeping[0x00007f1edc650000]
   java.lang.Thread.State: RUNNABLE
	at com.sun.codemodel.JStringLiteral.generate(JStringLiteral.java:61)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:363)
	at com.sun.codemodel.JInvocation.generate(JInvocation.java:185)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JThrow.state(JThrow.java:67)
	at com.sun.codemodel.JFormatter.s(JFormatter.java:386)
	at com.sun.codemodel.JBlock.generateBody(JBlock.java:448)
	at com.sun.codemodel.JBlock.generate(JBlock.java:436)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JConditional.state(JConditional.java:115)
	at com.sun.codemodel.JFormatter.s(JFormatter.java:386)
	at com.sun.codemodel.JBlock.generateBody(JBlock.java:448)
	at com.sun.codemodel.JBlock.generate(JBlock.java:436)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JBlock.state(JBlock.java:464)
	at com.sun.codemodel.JFormatter.s(JFormatter.java:386)
	at com.sun.codemodel.JBlock.generateBody(JBlock.java:448)
	at com.sun.codemodel.JBlock.generate(JBlock.java:436)
	at com.sun.codemodel.JFormatter.g(JFormatter.java:350)
	at com.sun.codemodel.JBlock.state(JBlock.java:464)
	at com.sun.codemodel.JFormatter.s(JFormatter.java:386)
	at com.sun.codemodel.JMethod.declare(JMethod.java:460)
	at com.sun.codemodel.JFormatter.d(JFormatter.java:376)
	at com.sun.codemodel.JDefinedClass.declareBody(JDefinedClass.java:815)
	at com.sun.codemodel.JDefinedClass.declare(JDefinedClass.java:788)
	at com.sun.codemodel.JFormatter.d(JFormatter.java:376)
	at com.sun.codemodel.JFormatter.write(JFormatter.java:406)
	at com.sun.codemodel.JPackage.build(JPackage.java:438)
	at com.sun.codemodel.JCodeModel.build(JCodeModel.java:311)
	at com.sun.codemodel.JCodeModel.build(JCodeModel.java:301)
	at org.apache.drill.exec.expr.CodeGenerator.generate(CodeGenerator.java:191)
	at org.apache.drill.exec.compile.CodeCompiler.createInstances(CodeCompiler.java:177)
	at org.apache.drill.exec.compile.CodeCompiler.createInstance(CodeCompiler.java:159)
	at org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:325)
	at org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:319)
	at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.generateNewWorker(MergeJoinBatch.java:382)
	at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:193)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
	at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
	at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
	at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:325)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
	at org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:140)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
	at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144)
	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
	at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
	at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
	at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
	at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x00000000e2398088> (a java.util.concurrent.ThreadPoolExecutor$Worker)

{code}

I got not the same jstack as was described in previous messages, but in this case, I can get
an infinite loop in MergeJoin and I think it relates to this issue.

> RecordIterator can sometimes restart a query on close
> -----------------------------------------------------
>
>                 Key: DRILL-5083
>                 URL: https://issues.apache.org/jira/browse/DRILL-5083
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Roman
>            Priority: Minor
>         Attachments: DrillOperatorErrorHandlingRedesign.pdf
>
>
> This one is very confusing...
> In a test with a MergeJoin and external sort, operators are stacked something like this:
> {code}
> Screen
> - MergeJoin
> - - External Sort
> ...
> {code}
> Using the injector to force a OOM in spill, the external sort threw a UserException up
the stack. This was handed by:
> {code}
> IteratorValidatorBatchIterator.next( )
> RecordIterator.clearInflightBatches( )
> RecordIterator.close( )
> MergeJoinBatch.close( )
> {code}
> Which does the following:
> {code}
>       // Check whether next() should even have been called in current state.
>       if (null != exceptionState) {
>         throw new IllegalStateException(
> {code}
> But, the exceptionState is set, so we end up throwing an IllegalStateException during
cleanup.
> Seems the code should agree: if {{next( )}} will be called during cleanup, then {{next(
)}} should gracefully handle that case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message