drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rahul Challapalli (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-4860) Partition pruning does not work when we have a correlated sub-query
Date Mon, 22 Aug 2016 18:46:20 GMT

     [ https://issues.apache.org/jira/browse/DRILL-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rahul Challapalli updated DRILL-4860:
-------------------------------------
    Attachment: l_3level.tgz

Contents of the log file
{code}
2016-08-22 18:43:10,348 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.drill.exec.work.foreman.Foreman
- Query text for query id 2844b840-84f0-eeb8-e8a9-a7697f9f0043: explain plan for select count
(*) from (
  select t1.l_orderkey, t1.l_linenumber from dfs.`/drill/testdata/partition_pruning/nested/l_3level`
t1
  where
    t1.dir0 = 1 and t1.dir1='one' and t1.dir2 = '2015-7-12' and t1.l_orderkey in (
      select t2.l_orderkey from dfs.`/drill/testdata/partition_pruning/nested/l_3level` t2
where t2.dir0 = 1 and t2.dir1='two' and t2.dir2 = '2015-8-12' and t2.l_discount = t1.l_discount
    )
) data
2016-08-22 18:43:10,544 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Beginning partition pruning, pruning class: org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnProjectRule
2016-08-22 18:43:10,545 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Total elapsed time to build and analyze filter tree: 0 ms
2016-08-22 18:43:10,545 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Elapsed time to populate partitioning column vectors: 0 ms within batchIndex: 0
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Elapsed time in interpreter evaluation: 1 ms within batchIndex: 0 with # of partitions :
11
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Pruned 11 partitions down to 1
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Total pruning elapsed time: 2 ms
2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Beginning partition pruning, pruning class: org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule

2016-08-22 18:43:10,547 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Total elapsed time to build and analyze filter tree: 0 ms
2016-08-22 18:43:10,548 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Elapsed time to populate partitioning column vectors: 0 ms within batchIndex: 0
2016-08-22 18:43:10,549 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Elapsed time in interpreter evaluation: 1 ms within batchIndex: 0 with # of partitions :
11
2016-08-22 18:43:10,549 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Pruned 11 partitions down to 1 
2016-08-22 18:43:10,549 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Total pruning elapsed time: 1 ms
2016-08-22 18:43:10,561 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Took 0 ms to get file statuses
2016-08-22 18:43:10,563 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.914413ms
avg, 1ms max.
2016-08-22 18:43:10,563 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Earliest start: 0.326000 μs,
Latest start: 0.326000 μs, Average start: 0.326000 μs .
2016-08-22 18:43:10,563 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Took 2 ms to read file metadata
2016-08-22 18:43:10,577 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Fetch parquet metadata: Executed 11 out of 11 using 11 threads. Time: 4ms total, 2.500455ms
avg, 2ms max.
2016-08-22 18:43:10,577 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Fetch parquet metadata: Executed 11 out of 11 using 11 threads. Earliest start: 208.722000
μs, Latest start: 1811.004000 μs, Average start: 1085.901091 μs .
2016-08-22 18:43:10,580 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Took 0 ms to get file statuses
2016-08-22 18:43:10,582 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.589404ms
avg, 1ms max.
2016-08-22 18:43:10,582 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Earliest start: 0.205000 μs,
Latest start: 0.205000 μs, Average start: 0.205000 μs .
2016-08-22 18:43:10,582 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Took 1 ms to read file metadata
2016-08-22 18:43:10,993 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2844b840-84f0-eeb8-e8a9-a7697f9f0043:0:0: State change requested AWAITING_ALLOCATION -->
RUNNING
2016-08-22 18:43:10,993 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 2844b840-84f0-eeb8-e8a9-a7697f9f0043:0:0: State to report: RUNNING
2016-08-22 18:43:11,000 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2844b840-84f0-eeb8-e8a9-a7697f9f0043:0:0: State change requested RUNNING --> FINISHED
2016-08-22 18:43:11,001 [2844b840-84f0-eeb8-e8a9-a7697f9f0043:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 2844b840-84f0-eeb8-e8a9-a7697f9f0043:0:0: State to report: FINISHED
{code}

> Partition pruning does not work when we have a correlated sub-query
> -------------------------------------------------------------------
>
>                 Key: DRILL-4860
>                 URL: https://issues.apache.org/jira/browse/DRILL-4860
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>    Affects Versions: 1.8.0
>            Reporter: Rahul Challapalli
>         Attachments: l_3level.tgz
>
>
> git commit # : 0a4c21cc15329c063f256f6fbf2c4c69a90d9fa1
> The below query indicates that partition pruning did not occur for the sub-query
> {code}
> explain plan for select count (*) from (
>   select t1.l_orderkey, t1.l_linenumber from dfs.`/drill/testdata/partition_pruning/nested/l_3level`
t1
>   where
>     t1.dir0 = 1 and t1.dir1='one' and t1.dir2 = '2015-7-12' and t1.l_orderkey in (
>       select t2.l_orderkey from dfs.`/drill/testdata/partition_pruning/nested/l_3level`
t2 where t2.dir0 = 1 and t2.dir1='two' and t2.dir2 = '2015-8-12' and t2.l_discount > t1.l_discount
>     )
> ) data;
> +------+------+
> | text | json |
> +------+------+
> | 00-00    Screen
> 00-01      Project(EXPR$0=[$0])
> 00-02        StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03          Project($f0=[0])
> 00-04            Project(l_discount=[$2], l_orderkey=[$3], l_orderkey0=[$0], l_discount0=[$1])
> 00-05              HashJoin(condition=[AND(=($2, $1), =($3, $0))], joinType=[inner])
> 00-07                HashAgg(group=[{0, 1}])
> 00-09                  Project(l_orderkey=[$0], l_discount0=[$2])
> 00-10                    Project(l_orderkey=[$1], l_discount=[$2], l_discount0=[$0])
> 00-11                      HashJoin(condition=[=($2, $0)], joinType=[inner])
> 00-13                        HashAgg(group=[{0}])
> 00-15                          Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/two/2015-8-12/30.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/three/2015-7-12/40.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-13/20.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-12/10.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-9-12/50.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-7-12/30.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-8-12/40.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/three/2015-8-12/10.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-9-12/20.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-7-12/50.parquet],
ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-8-12/10.parquet]],
selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level, numFiles=11, usedMetadataFile=false,
columns=[`l_discount`]]])
> 00-12                        Project(l_orderkey=[$0], l_discount0=[$1])
> 00-14                          Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/two/2015-8-12/30.parquet]],
selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level, numFiles=1, usedMetadataFile=false,
columns=[`l_orderkey`, `l_discount`]]])
> 00-06                Project(l_discount=[$0], l_orderkey0=[$1])
> 00-08                  Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-12/10.parquet]],
selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level, numFiles=1, usedMetadataFile=false,
columns=[`l_discount`, `l_orderkey`]]])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message