drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Hou (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-5778) Drill seems to run out of memory but completes execution
Date Sat, 09 Sep 2017 00:34:00 GMT
Robert Hou created DRILL-5778:
---------------------------------

             Summary: Drill seems to run out of memory but completes execution
                 Key: DRILL-5778
                 URL: https://issues.apache.org/jira/browse/DRILL-5778
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Relational Operators
    Affects Versions: 1.11.0
            Reporter: Robert Hou
            Assignee: Paul Rogers
             Fix For: 1.12.0


Query is:
{noformat}
ALTER SESSION SET `exec.sort.disable_managed` = false;
alter session set `planner.width.max_per_node` = 1;
alter session set `planner.disable_exchanges` = true;
alter session set `planner.width.max_per_query` = 1;
alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
select count(*) from (select * from (select id, flatten(str_list) str from dfs.`/drill/testdata/resource-manager/flatten-large-small.json`)
d order by d.str) d1 where d1.id=0;
{noformat}

Plan is:
{noformat}
| 00-00    Screen
00-01      Project(EXPR$0=[$0])
00-02        StreamAgg(group=[{}], EXPR$0=[$SUM0($0)])
00-03          UnionExchange
01-01            StreamAgg(group=[{}], EXPR$0=[COUNT()])
01-02              Project($f0=[0])
01-03                SelectionVectorRemover
01-04                  Filter(condition=[=($0, 0)])
01-05                    SingleMergeExchange(sort0=[1 ASC])
02-01                      SelectionVectorRemover
02-02                        Sort(sort0=[$1], dir0=[ASC])
02-03                          Project(id=[$0], str=[$1])
02-04                            HashToRandomExchange(dist0=[[$1]])
03-01                              UnorderedMuxExchange
04-01                                Project(id=[$0], str=[$1], E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($1,
1301011)])
04-02                                  Flatten(flattenField=[$1])
04-03                                    Project(id=[$0], str=[$1])
04-04                                      Scan(groupscan=[EasyGroupScan [selectionRoot=maprfs:/drill/testdata/resource-manager/flatten-large-small.json,
numFiles=1, columns=[`id`, `str_list`], files=[maprfs:///drill/testdata/resource-manager/flatten-large-small.json]]])
{noformat}

>From drillbit.log:
{noformat}
2017-09-08 05:07:21,515 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Actual batch schema & sizes {
  str(type: REQUIRED VARCHAR, count: 4096, std size: 54, actual size: 134, data size: 548360)
  id(type: OPTIONAL BIGINT, count: 4096, std size: 8, actual size: 9, data size: 36864)
  Records: 4096, Total size: 1073819648, Data size: 585224, Gross row width: 262163, Net row
width: 143, Density: 1}
2017-09-08 05:07:21,515 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] ERROR o.a.d.e.p.i.x.m.ExternalSortBatch
- Insufficient memory to merge two batches. Incoming batch size: 1073819648, available memory:
2147483648
2017-09-08 05:07:21,517 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] INFO  o.a.d.e.c.ClassCompilerSelector
- Java compiler policy: DEFAULT, Debug option: true
2017-09-08 05:07:21,517 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.compile.JaninoClassCompiler
- Compiling (source size=3.3 KiB):

...

2017-09-08 05:07:21,536 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.exec.compile.ClassTransformer
- Compiled and merged SingleBatchSorterGen2677: bytecode size = 3.6 KiB, time = 19 ms.
2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.t.g.SingleBatchSorterGen2677
- Took 5608 us to sort 4096 records
2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Input Batch Estimates: record size = 143 bytes; net = 1073819648 bytes, gross = 1610729472,
records = 4096
2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Spill batch size: net = 1048476 bytes, gross = 1572714 bytes, records = 7332; spill file
= 268435456 bytes
2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Output batch size: net = 9371505 bytes, gross = 14057257 bytes, records = 65535
2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Available memory: 2147483648, buffer memory = 2143289744, merge memory = 2128740638
2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.t.g.SingleBatchSorterGen2677
- Took 4303 us to sort 4096 records
2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Input Batch Estimates: record size = 266 bytes; net = 1073819648 bytes, gross = 1610729472,
records = 4096
2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Spill batch size: net = 1048572 bytes, gross = 1572858 bytes, records = 3942; spill file
= 268435456 bytes
2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Output batch size: net = 16777152 bytes, gross = 25165728 bytes, records = 63072
2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG o.a.d.e.p.i.x.m.ExternalSortBatch
- Available memory: 2147483648, buffer memory = 2143289360, merge memory = 2113929344
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message