hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikram Dixit Kumaraswamy" <vikram.di...@gmail.com>
Subject Re: Review Request 16728: Implement non-staged MapJoin
Date Mon, 27 Jan 2014 21:52:02 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16728/#review32885
-----------------------------------------------------------



itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
<https://reviews.apache.org/r/16728/#comment61876>

    This would break tez tests.



itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
<https://reviews.apache.org/r/16728/#comment61875>

    This would eliminate tez unit tests. Was this intentional?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java
<https://reviews.apache.org/r/16728/#comment61880>

    Could you raise a jira for this.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java
<https://reviews.apache.org/r/16728/#comment61882>

    can it be only these 2 operators? Maybe common join operator can be used?


- Vikram Dixit Kumaraswamy


On Jan. 20, 2014, 5 a.m., Navis Ryu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16728/
> -----------------------------------------------------------
> 
> (Updated Jan. 20, 2014, 5 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-6144
>     https://issues.apache.org/jira/browse/HIVE-6144
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> For map join, all data in small aliases are hashed and stored into temporary file in
MapRedLocalTask. But for some aliases without filter or projection, it seemed not necessary
to do that. For example.
> 
> {noformat}
> select a.* from src a join src b on a.key=b.key;
> {noformat}
> 
> makes plan like this.
> {noformat}
> STAGE PLANS:
>   Stage: Stage-4
>     Map Reduce Local Work
>       Alias -> Map Local Tables:
>         a 
>           Fetch Operator
>             limit: -1
>       Alias -> Map Local Operator Tree:
>         a 
>           TableScan
>             alias: a
>             HashTable Sink Operator
>               condition expressions:
>                 0 {key} {value}
>                 1 
>               handleSkewJoin: false
>               keys:
>                 0 [Column[key]]
>                 1 [Column[key]]
>               Position of Big Table: 1
> 
>   Stage: Stage-3
>     Map Reduce
>       Alias -> Map Operator Tree:
>         b 
>           TableScan
>             alias: b
>             Map Join Operator
>               condition map:
>                    Inner Join 0 to 1
>               condition expressions:
>                 0 {key} {value}
>                 1 
>               handleSkewJoin: false
>               keys:
>                 0 [Column[key]]
>                 1 [Column[key]]
>               outputColumnNames: _col0, _col1
>               Position of Big Table: 1
>               Select Operator
>                 File Output Operator
>       Local Work:
>         Map Reduce Local Work
>   Stage: Stage-0
>     Fetch Operator
> {noformat}
> 
> table src(a) is fetched and stored as-is in MRLocalTask. With this patch, plan can be
like below.
> {noformat}
>   Stage: Stage-3
>     Map Reduce
>       Alias -> Map Operator Tree:
>         b 
>           TableScan
>             alias: b
>             Map Join Operator
>               condition map:
>                    Inner Join 0 to 1
>               condition expressions:
>                 0 {key} {value}
>                 1 
>               handleSkewJoin: false
>               keys:
>                 0 [Column[key]]
>                 1 [Column[key]]
>               outputColumnNames: _col0, _col1
>               Position of Big Table: 1
>               Select Operator
>                   File Output Operator
>       Local Work:
>         Map Reduce Local Work
>           Alias -> Map Local Tables:
>             a 
>               Fetch Operator
>                 limit: -1
>           Alias -> Map Local Operator Tree:
>             a 
>               TableScan
>                 alias: a
>           Has Any Stage Alias: false
>   Stage: Stage-0
>     Fetch Operator
> {noformat}
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a78b72f 
>   conf/hive-default.xml.template 7cd8a1f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 9ad5986 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/MapJoinCounterHook.java 1b0d57e

>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java d8f4eb4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java a080fcc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java fc08b28 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 1e0314d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java bdc85b9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 56676df 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TemporaryHashSinkOperator.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 22e5777 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java 58484af 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 2d2508d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 2df8ab9 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java
5a53e15 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java 83b8d6e

>   ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java ebccb14

>   ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java c30da56 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 709c50e 
>   ql/src/test/queries/clientpositive/auto_join_without_localtask.q PRE-CREATION 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 0595cd6 
>   ql/src/test/results/clientnegative/deletejar.q.out b873e34 
>   ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out fa261b3 
>   ql/src/test/results/clientnegative/sortmerge_mapjoin_mismatch_1.q.out bca069a 
>   ql/src/test/results/clientpositive/alter_partition_coltype.q.out 685bf88 
>   ql/src/test/results/clientpositive/auto_join1.q.out b93c10f 
>   ql/src/test/results/clientpositive/auto_join15.q.out 6bfcfc7 
>   ql/src/test/results/clientpositive/auto_join17.q.out 698270e 
>   ql/src/test/results/clientpositive/auto_join19.q.out 247b081 
>   ql/src/test/results/clientpositive/auto_join2.q.out 756cfa1 
>   ql/src/test/results/clientpositive/auto_join21.q.out 053475f 
>   ql/src/test/results/clientpositive/auto_join22.q.out 6e5ae4c 
>   ql/src/test/results/clientpositive/auto_join24.q.out dbecb2c 
>   ql/src/test/results/clientpositive/auto_join25.q.out d6d0c90 
>   ql/src/test/results/clientpositive/auto_join26.q.out 7ca6bb9 
>   ql/src/test/results/clientpositive/auto_join28.q.out 488e15c 
>   ql/src/test/results/clientpositive/auto_join29.q.out 7798a0e 
>   ql/src/test/results/clientpositive/auto_join3.q.out 4279e8c 
>   ql/src/test/results/clientpositive/auto_join30.q.out a46ea2a 
>   ql/src/test/results/clientpositive/auto_join31.q.out 4504426 
>   ql/src/test/results/clientpositive/auto_join32.q.out da4ef55 
>   ql/src/test/results/clientpositive/auto_join9.q.out d2589ed 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 8e11616 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 95e86e1 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out fd8e9cc 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out 3c20e0f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out 6189df6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out c00b519 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 42bab30 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out ec33acd 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out f8c164a 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 6ee49a7 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 3f2672e 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 4cfd022 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 7b9f811 
>   ql/src/test/results/clientpositive/bucket_map_join_1.q.out f052025 
>   ql/src/test/results/clientpositive/bucket_map_join_2.q.out eb51e23 
>   ql/src/test/results/clientpositive/bucketcontext_1.q.out 9bdc1ee 
>   ql/src/test/results/clientpositive/bucketcontext_2.q.out d5d67dc 
>   ql/src/test/results/clientpositive/bucketcontext_3.q.out b226246 
>   ql/src/test/results/clientpositive/bucketcontext_4.q.out c27d927 
>   ql/src/test/results/clientpositive/bucketcontext_5.q.out 819362f 
>   ql/src/test/results/clientpositive/bucketcontext_6.q.out d7b1ad7 
>   ql/src/test/results/clientpositive/bucketcontext_7.q.out 80749f0 
>   ql/src/test/results/clientpositive/bucketcontext_8.q.out b8a4d4f 
>   ql/src/test/results/clientpositive/bucketmapjoin1.q.out 8d24d6f 
>   ql/src/test/results/clientpositive/bucketmapjoin10.q.out 371af7d 
>   ql/src/test/results/clientpositive/bucketmapjoin11.q.out fcfa41f 
>   ql/src/test/results/clientpositive/bucketmapjoin12.q.out aa50cfe 
>   ql/src/test/results/clientpositive/bucketmapjoin13.q.out eceeade 
>   ql/src/test/results/clientpositive/bucketmapjoin2.q.out 1dfe3e7 
>   ql/src/test/results/clientpositive/bucketmapjoin3.q.out ba1962d 
>   ql/src/test/results/clientpositive/bucketmapjoin4.q.out a9c4bce 
>   ql/src/test/results/clientpositive/bucketmapjoin5.q.out 4812a21 
>   ql/src/test/results/clientpositive/bucketmapjoin7.q.out f8ea2e2 
>   ql/src/test/results/clientpositive/bucketmapjoin8.q.out 8e2be73 
>   ql/src/test/results/clientpositive/bucketmapjoin9.q.out 38b9c87 
>   ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 0139546 
>   ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out bf00114 
>   ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out 2cc1a0f 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out 0014891 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_3.q.out f487b28 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out 14d0b6a 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out e83ef59 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out b5828f0 
>   ql/src/test/results/clientpositive/correlationoptimizer1.q.out 3ef5cf2 
>   ql/src/test/results/clientpositive/correlationoptimizer3.q.out e633287 
>   ql/src/test/results/clientpositive/correlationoptimizer4.q.out c037d3c 
>   ql/src/test/results/clientpositive/correlationoptimizer5.q.out 02ba37a 
>   ql/src/test/results/clientpositive/correlationoptimizer6.q.out 557ee9e 
>   ql/src/test/results/clientpositive/correlationoptimizer7.q.out b1fe60a 
>   ql/src/test/results/clientpositive/create_like_view.q.out e0bacbc 
>   ql/src/test/results/clientpositive/explain_rearrange.q.out 47b1afb 
>   ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out effa6c8 
>   ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 5a5c788 
>   ql/src/test/results/clientpositive/join25.q.out 630d862 
>   ql/src/test/results/clientpositive/join26.q.out 9fca2c3 
>   ql/src/test/results/clientpositive/join27.q.out 0ad8bff 
>   ql/src/test/results/clientpositive/join28.q.out f50072c 
>   ql/src/test/results/clientpositive/join29.q.out 0e0bbb4 
>   ql/src/test/results/clientpositive/join30.q.out 924579c 
>   ql/src/test/results/clientpositive/join31.q.out 85b0196 
>   ql/src/test/results/clientpositive/join32.q.out 9e84d84 
>   ql/src/test/results/clientpositive/join32_lessSize.q.out 0f6f8ec 
>   ql/src/test/results/clientpositive/join33.q.out 9e84d84 
>   ql/src/test/results/clientpositive/join34.q.out 1e2aa91 
>   ql/src/test/results/clientpositive/join35.q.out 83f5ba2 
>   ql/src/test/results/clientpositive/join36.q.out 922ddef 
>   ql/src/test/results/clientpositive/join37.q.out db905bd 
>   ql/src/test/results/clientpositive/join_map_ppr.q.out 131e65c 
>   ql/src/test/results/clientpositive/join_reorder4.q.out b14ea08 
>   ql/src/test/results/clientpositive/join_star.q.out 9b1adbf 
>   ql/src/test/results/clientpositive/mapjoin1.q.out e8b4fa8 
>   ql/src/test/results/clientpositive/mapjoin_distinct.q.out c4dc9d2 
>   ql/src/test/results/clientpositive/mapjoin_hook.q.out 8da5168 
>   ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 2c9f7f6 
>   ql/src/test/results/clientpositive/mapjoin_subquery.q.out e90a50a 
>   ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 220bf322 
>   ql/src/test/results/clientpositive/mapjoin_test_outer.q.out d66e079 
>   ql/src/test/results/clientpositive/merge3.q.out 6583b35 
>   ql/src/test/results/clientpositive/multiMapJoin1.q.out 624b47f 
>   ql/src/test/results/clientpositive/multiMapJoin2.q.out 6d0c7d2 
>   ql/src/test/results/clientpositive/multi_join_union.q.out be6e172 
>   ql/src/test/results/clientpositive/nullformatCTAS.q.out 54d7432 
>   ql/src/test/results/clientpositive/partition_type_check.q.out b166587 
>   ql/src/test/results/clientpositive/pcr.q.out 50f3bf5 
>   ql/src/test/results/clientpositive/ppd_union_view.q.out 3c86758 
>   ql/src/test/results/clientpositive/push_or.q.out 879f2c0 
>   ql/src/test/results/clientpositive/reduce_deduplicate_exclude_join.q.out fdde8df 
>   ql/src/test/results/clientpositive/sample10.q.out c7be38d 
>   ql/src/test/results/clientpositive/sample8.q.out ae1f56a 
>   ql/src/test/results/clientpositive/select_transform_hint.q.out d9ca1fc 
>   ql/src/test/results/clientpositive/show_create_table_alter.q.out 69a9962 
>   ql/src/test/results/clientpositive/show_create_table_serde.q.out 811ae4a 
>   ql/src/test/results/clientpositive/show_functions.q.out 57c9036 
>   ql/src/test/results/clientpositive/skewjoin.q.out 2e03b7e 
>   ql/src/test/results/clientpositive/skewjoin_noskew.q.out fe8da1e 
>   ql/src/test/results/clientpositive/smb_mapjoin_13.q.out b7dc7f2 
>   ql/src/test/results/clientpositive/smb_mapjoin_15.q.out 9adb0d0 
>   ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out 2011bc3 
>   ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out 3f2f6ab 
>   ql/src/test/results/clientpositive/stats11.q.out 44ddc03 
>   ql/src/test/results/clientpositive/subq_where_serialization.q.out 94840c4 
>   ql/src/test/results/clientpositive/subquery_in_having.q.out b293ed8 
>   ql/src/test/results/clientpositive/transform_ppr1.q.out bb7ca29 
>   ql/src/test/results/clientpositive/transform_ppr2.q.out 33fe33e 
>   ql/src/test/results/clientpositive/union_ppr.q.out 352fa4b 
>   ql/src/test/results/clientpositive/unset_table_view_property.q.out 2aa1dae 
>   ql/src/test/results/clientpositive/vector_left_outer_join.q.out 0113bb5 
>   ql/src/test/results/clientpositive/vectorized_context.q.out 5cd9f61 
>   ql/src/test/results/clientpositive/vectorized_mapjoin.q.out 13d5489 
> 
> Diff: https://reviews.apache.org/r/16728/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message