hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yin Huai" <h...@cse.ohio-state.edu>
Subject Re: Review Request: HIVE-2206: add a new optimizer for query correlation discovery and optimization
Date Mon, 19 Nov 2012 19:51:17 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7126/
-----------------------------------------------------------

(Updated Nov. 19, 2012, 7:51 p.m.)


Review request for hive.


Changes
-------

Correlation optimizer will guess which join operators at the bottom (input tables are not
intermediate tables) will be optimized by auto join convert and ignore those join operators
in the optimization of correlation optimizer.


Description
-------

This optimizer exploits intra-query correlations and merges multiple correlated MapReduce
jobs into one jobs. Open a new request since I have been working on hive-git.


This addresses bug HIVE-2206.
    https://issues.apache.org/jira/browse/HIVE-2206


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9fa9525 
  conf/hive-default.xml.template f332f3a 
  ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 7c4c413

  ql/src/java/org/apache/hadoop/hive/ql/exec/BaseReduceSinkOperator.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationCompositeOperator.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationLocalSimulativeReduceSinkOperator.java
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationReducerDispatchOperator.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/ExecReducer.java 18a9bd2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 46daeb2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 68302f8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 0c22141 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 919a140 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 1469325 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationOptimizer.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationOptimizerUtils.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java edde378 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java d1555e2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 2bf284d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 330aa52 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseReduceSinkDesc.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationCompositeDesc.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationLocalSimulativeReduceSinkDesc.java
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationReducerDispatchDesc.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 5a9f064 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java b33d616 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 9a95efd 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 6f8bc47 
  ql/src/test/queries/clientpositive/correlationoptimizer1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/correlationoptimizer2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/correlationoptimizer3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/correlationoptimizer4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/correlationoptimizer5.q PRE-CREATION 
  ql/src/test/results/clientpositive/correlationoptimizer1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/correlationoptimizer2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/correlationoptimizer3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/correlationoptimizer4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/correlationoptimizer5.q.out PRE-CREATION 
  ql/src/test/results/compiler/plan/groupby1.q.xml cd0d6e4 
  ql/src/test/results/compiler/plan/groupby2.q.xml 7b07f02 
  ql/src/test/results/compiler/plan/groupby3.q.xml a6a1986 
  ql/src/test/results/compiler/plan/groupby5.q.xml 25e3583 

Diff: https://reviews.apache.org/r/7126/diff/


Testing
-------

All tests pass.


Thanks,

Yin Huai


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message