drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Girish (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-3555) Changing defaults for planner.memory.max_query_memory_per_node causes queries with window function to fail
Date Sat, 25 Jul 2015 00:51:04 GMT
Abhishek Girish created DRILL-3555:
--------------------------------------

             Summary: Changing defaults for planner.memory.max_query_memory_per_node causes
queries with window function to fail
                 Key: DRILL-3555
                 URL: https://issues.apache.org/jira/browse/DRILL-3555
             Project: Apache Drill
          Issue Type: Bug
          Components: Query Planning & Optimization
    Affects Versions: 1.1.0, 1.2.0
         Environment: 4 Nodes. Direct Memory= 48 GB each
            Reporter: Abhishek Girish
            Assignee: Jinfeng Ni
            Priority: Critical


Changing the default value for planner.memory.max_query_memory_per_node from 2 GB to anything
higher causes queries with window functions to fail. 

Changed system options
{code:sql}
> select * from sys.options where status like '%CHANGE%';
+-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
|                   name                    |   kind   |  type   |  status  |   num_val  
| string_val  | bool_val  | float_val  |
+-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
| planner.enable_decimal_data_type          | BOOLEAN  | SYSTEM  | CHANGED  | null       
| null        | true      | null       |
| planner.memory.max_query_memory_per_node  | LONG     | SYSTEM  | CHANGED  | 8589934592 
| null        | null      | null       |
+-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
2 rows selected (0.249 seconds)
{code}

Query
{code:sql}
> SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) FROM store_sales
ss LIMIT 20;

java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException: Adding
this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500,
Incoming batchBytes 127875. maxBytes 1073741824
Fragment 1:0
[Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
at sqlline.SqlLine.print(SqlLine.java:1583)
at sqlline.Commands.execute(Commands.java:852)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:738)
at sqlline.SqlLine.begin(SqlLine.java:612)
at sqlline.SqlLine.start(SqlLine.java:366)
at sqlline.SqlLine.main(SqlLine.java:259)
{code}


Log:
{code}
2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State change requested RUNNING --> FINISHED

2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State to report: FINISHED

2015-07-23 18:17:05,485 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR o.a.d.e.p.i.s.SortRecordBatchBuilder
- Adding this batch causes the total size to exceed max allowed size. Current runningBytes
1073638500, Incoming batchBytes 127875. maxBytes 1073741824

2015-07-23 18:17:05,486 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested RUNNING --> FAILED

...

2015-07-23 18:17:05,990 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested FAILED --> FINISHED

2015-07-23 18:17:05,999 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR o.a.d.e.w.fragment.FragmentExecutor
- SYSTEM ERROR: DrillRuntimeException: Adding this batch causes the total size to exceed max
allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824

Fragment 1:0

[Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]

org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: DrillRuntimeException: Adding
this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500,
Incoming batchBytes 127875. maxBytes 1073741824

Fragment 1:0

[Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]

        at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

        at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_65]

        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]

Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Adding this batch causes
the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes
127875. maxBytes 1073741824

 at org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:115)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

        at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:380)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

       ...

2015-07-23 18:17:06,021 [BitServer-3] WARN  o.a.d.e.w.b.ControlMessageHandler - Dropping request
to cancel fragment. 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0 does not exist.

2015-07-23 18:17:06,022 [BitServer-3] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4:
State change requested RUNNING --> CANCELLATION_REQUESTED


2015-07-23 18:17:06,023 [BitServer-3] INFO  o.a.d.e.w.f.FragmentStatusReporter - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4:
State to report: CANCELLATION_REQUESTED
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message