hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Sun" <chao....@cloudera.com>
Subject Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]
Date Thu, 16 Oct 2014 01:25:28 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 16, 2014, 1:25 a.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Addressing the comments. Also, I'm thinking about adding another test for multi-insert in
another JIRA, specifically check if the plan after splitting is in the correct shape.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the
work is connected to multiple child works. The way splitting the operator tree is performed
by cloning the original work and removing unwanted branches in the operator tree. Please refer
to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method
that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such
information will be needed during SparkPlan generation (HIVE-8437).


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc

  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344

  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
  ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
  ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
  ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
  ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out
fd477ca 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
  ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
  ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------


Thanks,

Chao Sun


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message