hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Dere" <jd...@hortonworks.com>
Subject Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez
Date Fri, 10 Jul 2015 22:17:49 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/
-----------------------------------------------------------

(Updated July 10, 2015, 10:17 p.m.)


Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.


Changes
-------

- Rebase with trunk, looks like some methods in GenTezUtils were converted to static
- When selecting distributed hash join, the join operator should get OpTraits/stats set
- For the issue regarding the flattened expressions in the vectorized rowObjectInspector,
change the workaround to un-flatten the object inspector during JoinUtil.getObjectInspectorsFromEvaluators().
This is still a bit of a workaround, but only requires a change in 1 place, rather than the
2 changes needed in the previous solution (having to modify the column names during vectorized
MapJoinOperator, as well as when generating the vectorized rowObjectInspector in VectorizedBatchUtil)
- In the reducer, only the big table's input source should be vectorized


Bugs: HIVE-10673
    https://issues.apache.org/jira/browse/HIVE-10673


Repository: hive-git


Description
-------

Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the reducer are unsorted.


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 39477d6 
  itests/src/test/resources/testconfiguration.properties 97715fc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 7b57550 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 15cafdd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 545d7c6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 7d79e87 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java e9bd44a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
4c8c4b1 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 5a87bd6 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 4d84f0f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java bca91dd 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java f474eae 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 93ad145 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6b3e19d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java fa697ef 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java f9c34cb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java a78a92e 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_2.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out PRE-CREATION

  ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_2.q.out PRE-CREATION


Diff: https://reviews.apache.org/r/34059/diff/


Testing
-------

q-file tests added


Thanks,

Jason Dere


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message