hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Sun" <chao....@cloudera.com>
Subject Review Request 26001: HIVE-8233 - multi-table insertion doesn't work with ForwardOperator [Spark Branch]
Date Wed, 24 Sep 2014 19:25:59 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26001/
-----------------------------------------------------------

Review request for hive, Brock Noland and Xuefu Zhang.


Bugs: hive-8233
    https://issues.apache.org/jira/browse/hive-8233


Repository: hive-git


Description
-------

Right now, for multi-table insertion, we will start from multiple FileSinkOperators, and break
from their lowest common ancestor, adding temporary FileSinkOperator and TableScanOperators.
A special case is when the LCA is a ForwardOperator, in which case we don't break it, since
it's already been optimized.
However, there's a issue, considering the following plan:
      ...
      RS_0
       |
      FOR
       |
     /   \
   GBY_1  GBY_2
    |     |
   ...   ...
    |     |
   RS_1  RS_2
    |     |
   ...   ...
    |     |
   FS_1  FS_2
which may result to:
          RW
         /  \
       RW    RW
Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream branches will get
duplicated (and same) input.


Diffs
-----

  itests/src/test/resources/testconfiguration.properties 637fbc1 
  ql/src/test/queries/clientpositive/spark_groupby7_noskew_multi_single_reducer.q PRE-CREATION

  ql/src/test/queries/clientpositive/spark_groupby8.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_map.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_map_skew.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_noskew.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby9.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby_multi_insert_common_distinct.q PRE-CREATION

  ql/src/test/queries/clientpositive/spark_union17.q PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby7_noskew_multi_single_reducer.q.out
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_map.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_map_skew.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_noskew.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby9.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby_multi_insert_common_distinct.q.out
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_union17.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/26001/diff/


Testing
-------


Thanks,

Chao Sun


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message