hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yin Huai" <h...@cse.ohio-state.edu>
Subject Re: Review Request: HIVE-2206: add a new optimizer for query correlation discovery and optimization
Date Mon, 05 Dec 2011 19:12:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2001/
-----------------------------------------------------------

(Updated 2011-12-05 19:12:23.087778)


Review request for hive.


Changes
-------

CorrelationReduceSinkOperator has been merged into ReduceSinkOperator. Detailed comments has
been added to new operator.


Summary
-------

This optimizer exploits intra-query correlations and merges multiple correlated MapReduce
jobs into one jobs.


This addresses bug HIVE-2206.
    https://issues.apache.org/jira/browse/HIVE-2206


Diffs (updated)
-----

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationCompositeOperator.java PRE-CREATION

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationFakeReduceSinkOperator.java
PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationManualForwardOperator.java PRE-CREATION

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationReducerDispatchOperator.java
PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecReducer.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationOptimizer.java PRE-CREATION

  trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationOptimizerUtils.java PRE-CREATION

  trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationCompositeDesc.java PRE-CREATION

  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationFakeReduceSinkDesc.java PRE-CREATION

  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationManualForwardDesc.java PRE-CREATION

  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationReducerDispatchDesc.java PRE-CREATION

  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1210283 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 1210283 
  trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 1210283 
  trunk/ql/src/test/results/compiler/plan/groupby1.q.xml 1210283 
  trunk/ql/src/test/results/compiler/plan/groupby2.q.xml 1210283 
  trunk/ql/src/test/results/compiler/plan/groupby3.q.xml 1210283 
  trunk/ql/src/test/results/compiler/plan/groupby5.q.xml 1210283 

Diff: https://reviews.apache.org/r/2001/diff


Testing (updated)
-------

Previous version of diff passed all unit tests. Since the latest trunk (r1209696) cannot finish
all of unit tests, the latest version of diff has not been tested.


Thanks,

Yin


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message