Return-Path: X-Original-To: apmail-drill-issues-archive@minotaur.apache.org Delivered-To: apmail-drill-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9B7A618E4F for ; Fri, 4 Sep 2015 17:18:47 +0000 (UTC) Received: (qmail 70608 invoked by uid 500); 4 Sep 2015 17:18:46 -0000 Delivered-To: apmail-drill-issues-archive@drill.apache.org Received: (qmail 70498 invoked by uid 500); 4 Sep 2015 17:18:46 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 70303 invoked by uid 99); 4 Sep 2015 17:18:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Sep 2015 17:18:46 +0000 Date: Fri, 4 Sep 2015 17:18:46 +0000 (UTC) From: "Jacques Nadeau (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DRILL-3555) Changing defaults for planner.memory.max_query_memory_per_node causes queries with window function to fail MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/DRILL-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731088#comment-14731088 ] Jacques Nadeau commented on DRILL-3555: --------------------------------------- My bad. I didn't see the conversation and it wasn't linked here. I'm changing my -1 to a +1. thanks for the explanation. > Changing defaults for planner.memory.max_query_memory_per_node causes queries with window function to fail > ---------------------------------------------------------------------------------------------------------- > > Key: DRILL-3555 > URL: https://issues.apache.org/jira/browse/DRILL-3555 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization > Affects Versions: 1.1.0, 1.2.0 > Environment: 4 Nodes. Direct Memory= 48 GB each > Reporter: Abhishek Girish > Assignee: Deneche A. Hakim > Priority: Critical > Labels: window_function > Fix For: 1.2.0 > > > Changing the default value for planner.memory.max_query_memory_per_node from 2 GB to anything higher causes queries with window functions to fail. > Changed system options > {code:sql} > > select * from sys.options where status like '%CHANGE%'; > +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+ > | name | kind | type | status | num_val | string_val | bool_val | float_val | > +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+ > | planner.enable_decimal_data_type | BOOLEAN | SYSTEM | CHANGED | null | null | true | null | > | planner.memory.max_query_memory_per_node | LONG | SYSTEM | CHANGED | 8589934592 | null | null | null | > +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+ > 2 rows selected (0.249 seconds) > {code} > Query > {code:sql} > > SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) FROM store_sales ss LIMIT 20; > java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824 > Fragment 1:0 > [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010] > at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73) > at sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87) > at sqlline.TableOutputFormat.print(TableOutputFormat.java:118) > at sqlline.SqlLine.print(SqlLine.java:1583) > at sqlline.Commands.execute(Commands.java:852) > at sqlline.Commands.sql(Commands.java:751) > at sqlline.SqlLine.dispatch(SqlLine.java:738) > at sqlline.SqlLine.begin(SqlLine.java:612) > at sqlline.SqlLine.start(SqlLine.java:366) > at sqlline.SqlLine.main(SqlLine.java:259) > {code} > Log: > {code} > 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State change requested RUNNING --> FINISHED > 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO o.a.d.e.w.f.FragmentStatusReporter - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State to report: FINISHED > 2015-07-23 18:17:05,485 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR o.a.d.e.p.i.s.SortRecordBatchBuilder - Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824 > 2015-07-23 18:17:05,486 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested RUNNING --> FAILED > ... > 2015-07-23 18:17:05,990 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested FAILED --> FINISHED > 2015-07-23 18:17:05,999 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824 > Fragment 1:0 > [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824 > Fragment 1:0 > [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010] > at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824 > at org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:115) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:380) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... > 2015-07-23 18:17:06,021 [BitServer-3] WARN o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0 does not exist. > 2015-07-23 18:17:06,022 [BitServer-3] INFO o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4: State change requested RUNNING --> CANCELLATION_REQUESTED > 2015-07-23 18:17:06,023 [BitServer-3] INFO o.a.d.e.w.f.FragmentStatusReporter - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4: State to report: CANCELLATION_REQUESTED > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)