drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aman Sinha (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-918) Merge join results in a java.lang.IllegalArgumentException during sort
Date Thu, 05 Jun 2014 23:09:02 GMT

    [ https://issues.apache.org/jira/browse/DRILL-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019391#comment-14019391
] 

Aman Sinha edited comment on DRILL-918 at 6/5/14 11:08 PM:
-----------------------------------------------------------

The IllegalArgumentException is actually preceded by an AssertionError (this can be see if
assertions are enabled in the JVM arguments). 

Exception in thread "WorkManager-6" Exception in thread "WorkManager-5" java.lang.AssertionError
	at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:221)
	at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63)
	at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47)
	at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91)
	at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
	at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70)
	at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79)
	at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138)

The underlying assert is line 221 in ExternalSortBatch.java:   assert sv2.getCount() >
0;    
So we are getting an empty batch as input to the sort.  I believe this started happening after
we enabled the broadcast join.  The left child of the join is not distributed (previously
it was getting hash distributed) and one of the fragments encounters an empty batch.  


was (Author: amansinha100):
The IllegalArgumentException is actually preceded by an AssertionError (this can be see if
assertions are enabled in the jam arguments). 

Exception in thread "WorkManager-6" Exception in thread "WorkManager-5" java.lang.AssertionError
	at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:221)
	at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63)
	at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47)
	at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91)
	at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
	at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70)
	at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79)
	at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138)

The underlying assert is line 221 in ExternalSortBatch.java:   assert sv2.getCount() >
0;    
So we are getting an empty batch as input to the sort.  I believe this started happening after
we enabled the broadcast join.  The left child of the join is not distributed (previously
it was getting hash distributed) and one of the fragments encounters an empty batch.  

> Merge join results in a java.lang.IllegalArgumentException during sort
> ----------------------------------------------------------------------
>
>                 Key: DRILL-918
>                 URL: https://issues.apache.org/jira/browse/DRILL-918
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>            Reporter: Ramana Inukonda Nagaraj
>            Assignee: Aman Sinha
>
> This is one of the smoke tests which was passing earlier.
> The testcase looks like this
> alter session set `planner.enable_hashjoin` = false;
> select count(*)
>   from (select l.l_orderkey as x, c.c_custkey as y
>         from lineitem l
>                 left outer join customer c
>                         on l.l_orderkey = c.c_custkey) as foo
>   where x < 10000;
> And there is an error during the sort process(presumably because merge join is used)

> java.lang.IllegalArgumentException: Undefined for 0
>         at org.apache.hadoop.util.QuickSort.getMaxDepth(QuickSort.java:41) ~[hadoop-0.20.2-dev-core.jar:na]
>         at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) ~[hadoop-0.20.2-dev-core.jar:na]
>         at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:51) ~[hadoop-0.20.2-dev-core.jar:na]
>         at org.apache.drill.exec.test.generated.SingleBatchSorterGen50.sort(SingleBatchSorterTemplate.java:51)
~[na:na]
>         at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:225)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.next(StreamingAggBatch.java:86)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98)
~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
>         at java.lang.Thread.run(Thread.java:744) [na:1.7.0_4



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message