Return-Path: X-Original-To: apmail-drill-issues-archive@minotaur.apache.org Delivered-To: apmail-drill-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 17B42105BE for ; Fri, 6 Jun 2014 00:56:08 +0000 (UTC) Received: (qmail 72540 invoked by uid 500); 5 Jun 2014 23:09:28 -0000 Delivered-To: apmail-drill-issues-archive@drill.apache.org Received: (qmail 72517 invoked by uid 500); 5 Jun 2014 23:09:28 -0000 Mailing-List: contact issues-help@drill.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.incubator.apache.org Delivered-To: mailing list issues@drill.incubator.apache.org Received: (qmail 72504 invoked by uid 99); 5 Jun 2014 23:09:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2014 23:09:28 +0000 X-ASF-Spam-Status: No, hits=-2000.7 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 05 Jun 2014 23:09:27 +0000 Received: (qmail 72430 invoked by uid 99); 5 Jun 2014 23:09:02 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2014 23:09:02 +0000 Date: Thu, 5 Jun 2014 23:09:02 +0000 (UTC) From: "Aman Sinha (JIRA)" To: issues@drill.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (DRILL-918) Merge join results in a java.lang.IllegalArgumentException during sort MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/DRILL-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019391#comment-14019391 ] Aman Sinha edited comment on DRILL-918 at 6/5/14 11:08 PM: ----------------------------------------------------------- The IllegalArgumentException is actually preceded by an AssertionError (this can be see if assertions are enabled in the JVM arguments). Exception in thread "WorkManager-6" Exception in thread "WorkManager-5" java.lang.AssertionError at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:221) at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47) at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91) at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70) at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79) at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138) The underlying assert is line 221 in ExternalSortBatch.java: assert sv2.getCount() > 0; So we are getting an empty batch as input to the sort. I believe this started happening after we enabled the broadcast join. The left child of the join is not distributed (previously it was getting hash distributed) and one of the fragments encounters an empty batch. was (Author: amansinha100): The IllegalArgumentException is actually preceded by an AssertionError (this can be see if assertions are enabled in the jam arguments). Exception in thread "WorkManager-6" Exception in thread "WorkManager-5" java.lang.AssertionError at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:221) at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47) at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91) at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70) at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79) at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138) The underlying assert is line 221 in ExternalSortBatch.java: assert sv2.getCount() > 0; So we are getting an empty batch as input to the sort. I believe this started happening after we enabled the broadcast join. The left child of the join is not distributed (previously it was getting hash distributed) and one of the fragments encounters an empty batch. > Merge join results in a java.lang.IllegalArgumentException during sort > ---------------------------------------------------------------------- > > Key: DRILL-918 > URL: https://issues.apache.org/jira/browse/DRILL-918 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization > Reporter: Ramana Inukonda Nagaraj > Assignee: Aman Sinha > > This is one of the smoke tests which was passing earlier. > The testcase looks like this > alter session set `planner.enable_hashjoin` = false; > select count(*) > from (select l.l_orderkey as x, c.c_custkey as y > from lineitem l > left outer join customer c > on l.l_orderkey = c.c_custkey) as foo > where x < 10000; > And there is an error during the sort process(presumably because merge join is used) > java.lang.IllegalArgumentException: Undefined for 0 > at org.apache.hadoop.util.QuickSort.getMaxDepth(QuickSort.java:41) ~[hadoop-0.20.2-dev-core.jar:na] > at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) ~[hadoop-0.20.2-dev-core.jar:na] > at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:51) ~[hadoop-0.20.2-dev-core.jar:na] > at org.apache.drill.exec.test.generated.SingleBatchSorterGen50.sort(SingleBatchSorterTemplate.java:51) ~[na:na] > at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:225) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.next(StreamingAggBatch.java:86) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_4 -- This message was sent by Atlassian JIRA (v6.2#6252)