hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suhas Satish" <suhas.sat...@gmail.com>
Subject Re: Review Request 27265: Support SMB Join for Hive on Spark [Spark Branch]
Date Tue, 28 Oct 2014 05:28:36 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27265/#review58769
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/lib/TypeRule.java
<https://reviews.apache.org/r/27265/#comment99922>

    Is the cost being set to 1 so that it will be lower than other RuleRegExp.java based rules
and hence get triggered earlier? Since lower the cost, better the rule match.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java
<https://reviews.apache.org/r/27265/#comment99924>

    Can you change this to SMB Join to avoid confusion?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java
<https://reviews.apache.org/r/27265/#comment99925>

    again to avoid confusion, can you please rename the method to initSMBJoinPlan



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java
<https://reviews.apache.org/r/27265/#comment99926>

    so when deferSetup boolean flag is true, when does createMapWork get deferred to?


- Suhas Satish


On Oct. 28, 2014, 2:20 a.m., Szehon Ho wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27265/
> -----------------------------------------------------------
> 
> (Updated Oct. 28, 2014, 2:20 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> This change re-uses the SMBJoinOperator for Spark.  Background: the logical layer already
converts joins to SMB Joins.  This changes just introduces a class called "SparkSortMergeJoinFactory"
on the Spark-compile path which attaches the data structures (like local work, bucket info)
to the MapWork for the SMBJoinOperator to consume.  It is largely-based on the MapReduce class
"MapJoinFactory".
> 
> However, in spark-path, it is activated only for SMBJoin and not map-joins, as we have
another strategy for map-joins.  That is why there's a new optimizer-rule called "TypeRule",
so this processor is only run on SMBJoinOperators (which share same name with MapJoinOperators,
which is needed for logical-optimizers dealing with hints).
> 
> One major assumption around the whole SMB concept is that both tables have corresponding
buckets.  I found during testing of large numbers of buckets (like auto_sortmerge_join_16)
that "insert" into a bucketed table wasn't putting the same keys in corresponding buckets.
 I activated MR-style shuffle (hash-shuffle instead of total-order shuffle), and that seemed
to solve the issue.
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 00c9f4d 
>   ql/src/java/org/apache/hadoop/hive/ql/lib/TypeRule.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java ae1d1ab 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java ed88c60

>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 8e28887 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java 4f5feca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 1c663c4 
>   ql/src/test/results/clientpositive/spark/auto_join0.q.out 76ff63d 
>   ql/src/test/results/clientpositive/spark/auto_join10.q.out 05a5912 
>   ql/src/test/results/clientpositive/spark/auto_join11.q.out 998c28b 
>   ql/src/test/results/clientpositive/spark/auto_join12.q.out d2b7993 
>   ql/src/test/results/clientpositive/spark/auto_join13.q.out 78aa01e 
>   ql/src/test/results/clientpositive/spark/auto_join15.q.out 5916070 
>   ql/src/test/results/clientpositive/spark/auto_join16.q.out 0b6807d 
>   ql/src/test/results/clientpositive/spark/auto_join18.q.out 6083b38 
>   ql/src/test/results/clientpositive/spark/auto_join18_multi_distinct.q.out 01c8f0a 
>   ql/src/test/results/clientpositive/spark/auto_join20.q.out a8f2b9a 
>   ql/src/test/results/clientpositive/spark/auto_join21.q.out f9ac35d 
>   ql/src/test/results/clientpositive/spark/auto_join22.q.out 516322c 
>   ql/src/test/results/clientpositive/spark/auto_join23.q.out ce5a670 
>   ql/src/test/results/clientpositive/spark/auto_join24.q.out 15b8888 
>   ql/src/test/results/clientpositive/spark/auto_join27.q.out 67f5739 
>   ql/src/test/results/clientpositive/spark/auto_join28.q.out b979661 
>   ql/src/test/results/clientpositive/spark/auto_join29.q.out 0951b8d 
>   ql/src/test/results/clientpositive/spark/auto_join30.q.out 98b3974 
>   ql/src/test/results/clientpositive/spark/auto_join31.q.out df502c8 
>   ql/src/test/results/clientpositive/spark/auto_join32.q.out 8d83188 
>   ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out e64d4fb 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 9158d65 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_10.q.out f608cc5 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out 3c26363 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out 65e496f 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out a5a281b 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_14.q.out 2fc3bb6 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_15.q.out 74cbd7c 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out d1bb7a0 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out d57a1d7 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 8244c50 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 2ab1bca 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_6.q.out bc4a163 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out 16ef3ae 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out 9fd3e5a 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_9.q.out a7f994f 
>   ql/src/test/results/clientpositive/spark/bucket2.q.out b1b2997 
>   ql/src/test/results/clientpositive/spark/bucket3.q.out 019c11a 
>   ql/src/test/results/clientpositive/spark/bucket4.q.out 2cbab11 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_1.q.out 4ec619e 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_2.q.out 1c288c2 
>   ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out 8be3edd 
>   ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out 9e45843 
>   ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out 0c1ac4b 
>   ql/src/test/results/clientpositive/spark/bucketmapjoin13.q.out dc1b8cf 
>   ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out 6d72fdf 
>   ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out d80bdcf 
>   ql/src/test/results/clientpositive/spark/count.q.out c527c1d 
>   ql/src/test/results/clientpositive/spark/ctas.q.out 0ded266 
>   ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out 590b265

>   ql/src/test/results/clientpositive/spark/escape_clusterby1.q.out 52bdf6a 
>   ql/src/test/results/clientpositive/spark/escape_distributeby1.q.out 736db5e 
>   ql/src/test/results/clientpositive/spark/escape_orderby1.q.out 6e1c0cf 
>   ql/src/test/results/clientpositive/spark/escape_sortby1.q.out 58b663c 
>   ql/src/test/results/clientpositive/spark/groupby1.q.out 847f45c 
>   ql/src/test/results/clientpositive/spark/groupby10.q.out 2095843 
>   ql/src/test/results/clientpositive/spark/groupby11.q.out 70db5a5 
>   ql/src/test/results/clientpositive/spark/groupby2.q.out 86e2f2a 
>   ql/src/test/results/clientpositive/spark/groupby3.q.out 13a5fab 
>   ql/src/test/results/clientpositive/spark/groupby3_map.q.out dac2824 
>   ql/src/test/results/clientpositive/spark/groupby3_map_multi_distinct.q.out d2c054a

>   ql/src/test/results/clientpositive/spark/groupby3_map_skew.q.out ec6439a 
>   ql/src/test/results/clientpositive/spark/groupby3_noskew.q.out 0c9a7e1 
>   ql/src/test/results/clientpositive/spark/groupby3_noskew_multi_distinct.q.out 42fbb8c

>   ql/src/test/results/clientpositive/spark/groupby4.q.out 318c5a3 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 22a05b5 
>   ql/src/test/results/clientpositive/spark/groupby7_map_multi_single_reducer.q.out bc453c6

>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 2a07f2a 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 00a0707 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out
36640ef 
>   ql/src/test/results/clientpositive/spark/groupby8.q.out d8295ce 
>   ql/src/test/results/clientpositive/spark/groupby8_map.q.out b9aa597 
>   ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out b9aa597 
>   ql/src/test/results/clientpositive/spark/groupby8_noskew.q.out b9aa597 
>   ql/src/test/results/clientpositive/spark/groupby9.q.out bec2346 
>   ql/src/test/results/clientpositive/spark/groupby_complex_types.q.out 16fadea 
>   ql/src/test/results/clientpositive/spark/groupby_complex_types_multi_single_reducer.q.out
7470843 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 169c4ac 
>   ql/src/test/results/clientpositive/spark/groupby_multi_insert_common_distinct.q.out
d3457da 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 3abd0e3

>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer2.q.out 7f74c62

>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer3.q.out c4b7419

>   ql/src/test/results/clientpositive/spark/groupby_position.q.out 9e58189 
>   ql/src/test/results/clientpositive/spark/groupby_ppr.q.out 860aa58 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 0aeff6b 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 61dd2be 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 99da734 
>   ql/src/test/results/clientpositive/spark/having.q.out 5e9f20d 
>   ql/src/test/results/clientpositive/spark/input14.q.out e7d4db6 
>   ql/src/test/results/clientpositive/spark/input17.q.out 0882a29 
>   ql/src/test/results/clientpositive/spark/input18.q.out 802fb0a 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 33ecd07 
>   ql/src/test/results/clientpositive/spark/insert_into1.q.out e9be658 
>   ql/src/test/results/clientpositive/spark/insert_into2.q.out 5c8e9c7 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 6c0111d 
>   ql/src/test/results/clientpositive/spark/join0.q.out 55b725e 
>   ql/src/test/results/clientpositive/spark/join15.q.out 1651db1 
>   ql/src/test/results/clientpositive/spark/join18.q.out 7b64fb6 
>   ql/src/test/results/clientpositive/spark/join18_multi_distinct.q.out 57c4516 
>   ql/src/test/results/clientpositive/spark/join20.q.out f06ffac 
>   ql/src/test/results/clientpositive/spark/join21.q.out e81ec5a 
>   ql/src/test/results/clientpositive/spark/join23.q.out 3982ea7 
>   ql/src/test/results/clientpositive/spark/join29.q.out d5383d5 
>   ql/src/test/results/clientpositive/spark/join30.q.out 5c16622 
>   ql/src/test/results/clientpositive/spark/join31.q.out 9193df9 
>   ql/src/test/results/clientpositive/spark/join35.q.out 1750aec 
>   ql/src/test/results/clientpositive/spark/smb_mapjoin_17.q.out 482268c 
> 
> Diff: https://reviews.apache.org/r/27265/diff/
> 
> 
> Testing
> -------
> 
> Ran existing auto_sortmerge_* tests.
> 
> 
> Thanks,
> 
> Szehon Ho
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message