drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Girish (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3555) Changing defaults for planner.memory.max_query_memory_per_node causes queries with window function to fail
Date Mon, 27 Jul 2015 23:09:04 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14643581#comment-14643581
] 

Abhishek Girish commented on DRILL-3555:
----------------------------------------

As discussed offline, I'll skip running this query. In summary, this issue might have regressed.
But we never had thorough tests for them, so its also likely to be an existing issue which
was only observed recently. 

> Changing defaults for planner.memory.max_query_memory_per_node causes queries with window
function to fail
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: DRILL-3555
>                 URL: https://issues.apache.org/jira/browse/DRILL-3555
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>    Affects Versions: 1.1.0, 1.2.0
>         Environment: 4 Nodes. Direct Memory= 48 GB each
>            Reporter: Abhishek Girish
>            Assignee: Jinfeng Ni
>            Priority: Critical
>
> Changing the default value for planner.memory.max_query_memory_per_node from 2 GB to
anything higher causes queries with window functions to fail. 
> Changed system options
> {code:sql}
> > select * from sys.options where status like '%CHANGE%';
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> |                   name                    |   kind   |  type   |  status  |   num_val
  | string_val  | bool_val  | float_val  |
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> | planner.enable_decimal_data_type          | BOOLEAN  | SYSTEM  | CHANGED  | null  
     | null        | true      | null       |
> | planner.memory.max_query_memory_per_node  | LONG     | SYSTEM  | CHANGED  | 8589934592
 | null        | null      | null       |
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> 2 rows selected (0.249 seconds)
> {code}
> Query
> {code:sql}
> > SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) FROM store_sales
ss LIMIT 20;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException:
Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500,
Incoming batchBytes 127875. maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Log:
> {code}
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State change requested RUNNING --> FINISHED
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State to report: FINISHED
> 2015-07-23 18:17:05,485 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR o.a.d.e.p.i.s.SortRecordBatchBuilder
- Adding this batch causes the total size to exceed max allowed size. Current runningBytes
1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> 2015-07-23 18:17:05,486 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested RUNNING --> FAILED
> ...
> 2015-07-23 18:17:05,990 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested FAILED --> FINISHED
> 2015-07-23 18:17:05,999 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR o.a.d.e.w.fragment.FragmentExecutor
- SYSTEM ERROR: DrillRuntimeException: Adding this batch causes the total size to exceed max
allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: DrillRuntimeException:
Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500,
Incoming batchBytes 127875. maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
>         at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_65]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Adding this batch
causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming
batchBytes 127875. maxBytes 1073741824
>  at org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:115)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:380)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>        ...
> 2015-07-23 18:17:06,021 [BitServer-3] WARN  o.a.d.e.w.b.ControlMessageHandler - Dropping
request to cancel fragment. 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0 does not exist.
> 2015-07-23 18:17:06,022 [BitServer-3] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4:
State change requested RUNNING --> CANCELLATION_REQUESTED
> 2015-07-23 18:17:06,023 [BitServer-3] INFO  o.a.d.e.w.f.FragmentStatusReporter - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4:
State to report: CANCELLATION_REQUESTED
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message