hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-17087) Remove unnecessary HoS DPP trees during map-join conversion
Date Sun, 23 Jul 2017 22:30:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096693#comment-16096693
] 

Sahil Takiar edited comment on HIVE-17087 at 7/23/17 10:29 PM:
---------------------------------------------------------------

Here is a brief summary of the issue:

* When {{DynamicPartitionPruningOptimization}} is run, it runs before any join optimizations
are done (including map-join conversion)
* It's add a DPP subtree ({{SEL-GBY-SPARKPRUNINGSINK}} everywhere possible
* After DPP is run, there are a few places where a DPP subtree can be removed: {{SparkRemoveDynamicPruningBySize}}
and during {{SparkCompiler#runCycleAnalysisForPartitionPruning}}
* This patch adds another place where DPP can be removed, inside {{SparkMapJoinOptimizer}}
* There are certain scenarios where a DPP subtree needs to be removed during map-join conversion:
** Say there is a query that can take advantage of DPP, and the small table is partitioned
and the big table is un-partitioned
** If the join is converted to a map-join, it doesn't make sense to run DPP at all
*** Running DPP would require scanning the entire big table before the map-join even starts,
and would result in scanning the big table twice
* The example above can be extended to any scenario where DPP and map-joins are involved,
if there is a DPP subtree on the big table that is meant to prune partitions from the small
table, then the subtree should be completely removed
* Hive-on-Tez does the same thing, the majority of the code is in this patch is copied from
{{ConvertJoinMapJoin}}; the changes to {{ConvertJoinMapJoin}} were done in the original DPP
patch for Hive-on-Tez
* Some code was also copied from HIVE-10559, which fixes a NPE in this changes to the map-join
conversion
* Added a query to {{spark_dynamic_partition_pruning_2.q}} that was added to {{dynamic_partition_pruning_2.q}}
in HIVE-10559
* I added a new file called {{spark_dynamic_partition_pruning_3.q}} that adds some queries
that join two partitioned tables together, I couldn't find this style of query in the other
dpp .q files


was (Author: stakiar):
Here is a brief summary of the issue:

* When {{DynamicPartitionPruningOptimization}} is run, it runs before any join optimizations
are done (including map-join conversion)
* It's add a DPP subtree ({{SEL-GBY-SPARKPRUNINGSINK}} everywhere possible
* After DPP is run, there are a few places where a DPP subtree can be removed: {{SparkRemoveDynamicPruningBySize}}
and during {{SparkCompiler#runCycleAnalysisForPartitionPruning}}
* This patch adds another place where DPP can be removed, inside {{SparkMapJoinOptimizer}}
* There are certain scenarios where a DPP subtree needs to be removed during map-join conversion:
** Say there is a query that can take advantage of DPP, and the small table is partitioned
and the big table is un-partitioned
** If the join is converted to a map-join, it doesn't make sense to run DPP at all
*** Running DPP would require scanning the entire big table before the map-join even starts,
and would result in scanning the big table twic
* The example above can be extended to any scenario where DPP and map-joins are involved,
if there is a DPP subtree on the big table that is meant to prune partitions from the small
table, then the subtree should be completely removed
* Hive-on-Tez does the same thing, the majority of the code is in this patch is copied from
{{ConvertJoinMapJoin}}; the changes to {{ConvertJoinMapJoin}} were done in the original DPP
patch for Hive-on-Tez
* Some code was also copied from HIVE-10559, which fixes a NPE in this changes to the map-join
conversion
* Added a query to {{spark_dynamic_partition_pruning_2.q}} that was added to {{dynamic_partition_pruning_2.q}}
in HIVE-10559
* I added a new file called {{spark_dynamic_partition_pruning_3.q}} that adds some queries
that join two partitioned tables together, I couldn't find this style of query in the other
dpp .q files

> Remove unnecessary HoS DPP trees during map-join conversion
> -----------------------------------------------------------
>
>                 Key: HIVE-17087
>                 URL: https://issues.apache.org/jira/browse/HIVE-17087
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-17087.1.patch, HIVE-17087.2.patch
>
>
> Ran the following query in the {{TestSparkCliDriver}}:
> {code:sql}
> set hive.spark.dynamic.partition.pruning=true;
> set hive.auto.convert.join=true;
> create table partitioned_table1 (col int) partitioned by (part_col int);
> create table partitioned_table2 (col int) partitioned by (part_col int);
> create table regular_table (col int);
> insert into table regular_table values (1);
> alter table partitioned_table1 add partition (part_col = 1);
> insert into table partitioned_table1 partition (part_col = 1) values (1), (2), (3), (4),
(5), (6), (7), (8), (9), (10);
> alter table partitioned_table2 add partition (part_col = 1);
> insert into table partitioned_table2 partition (part_col = 1) values (1), (2), (3), (4),
(5), (6), (7), (8), (9), (10);
> explain select * from partitioned_table1, partitioned_table2 where partitioned_table1.part_col
= partitioned_table2.part_col;
> {code}
> and got the following explain plan:
> {code}
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-3 depends on stages: Stage-2
>   Stage-1 depends on stages: Stage-3
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
>     Spark
> #### A masked pattern was here ####
>       Vertices:
>         Map 3 
>             Map Operator Tree:
>                 TableScan
>                   alias: partitioned_table1
>                   Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column
stats: NONE
>                   Select Operator
>                     expressions: col (type: int), part_col (type: int)
>                     outputColumnNames: _col0, _col1
>                     Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column
stats: NONE
>                     Select Operator
>                       expressions: _col1 (type: int)
>                       outputColumnNames: _col0
>                       Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column
stats: NONE
>                       Group By Operator
>                         keys: _col0 (type: int)
>                         mode: hash
>                         outputColumnNames: _col0
>                         Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE
Column stats: NONE
>                         Spark Partition Pruning Sink Operator
>                           partition key expr: part_col
>                           Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE
Column stats: NONE
>                           target column name: part_col
>                           target work: Map 2
>   Stage: Stage-3
>     Spark
> #### A masked pattern was here ####
>       Vertices:
>         Map 2 
>             Map Operator Tree:
>                 TableScan
>                   alias: partitioned_table2
>                   Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column
stats: NONE
>                   Select Operator
>                     expressions: col (type: int), part_col (type: int)
>                     outputColumnNames: _col0, _col1
>                     Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column
stats: NONE
>                     Spark HashTable Sink Operator
>                       keys:
>                         0 _col1 (type: int)
>                         1 _col1 (type: int)
>             Local Work:
>               Map Reduce Local Work
>   Stage: Stage-1
>     Spark
> #### A masked pattern was here ####
>       Vertices:
>         Map 1 
>             Map Operator Tree:
>                 TableScan
>                   alias: partitioned_table1
>                   Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column
stats: NONE
>                   Select Operator
>                     expressions: col (type: int), part_col (type: int)
>                     outputColumnNames: _col0, _col1
>                     Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column
stats: NONE
>                     Map Join Operator
>                       condition map:
>                            Inner Join 0 to 1
>                       keys:
>                         0 _col1 (type: int)
>                         1 _col1 (type: int)
>                       outputColumnNames: _col0, _col1, _col2, _col3
>                       input vertices:
>                         1 Map 2
>                       Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE Column
stats: NONE
>                       File Output Operator
>                         compressed: false
>                         Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE
Column stats: NONE
>                         table:
>                             input format: org.apache.hadoop.mapred.SequenceFileInputFormat
>                             output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
>                             serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>             Local Work:
>               Map Reduce Local Work
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
>       Processor Tree:
>         ListSink
> {code}
> Stage-2 seems unnecessary, given that Stage-1 is going to do a full table scan of {{partitioned_table1}}
when running the map-join



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message