hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho" <sze...@cloudera.com>
Subject Re: Review Request 27247: convert joinOp to MapJoinOp and generate MapWorks only
Date Mon, 27 Oct 2014 23:03:59 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27247/#review58712
-----------------------------------------------------------


Hi Suhas, thanks a lot for this great work!

I haven't looked at all the details, but have some high-level comments just from the first
pass.  I think if we revert some of the SMB Join stuff and commented code that don't refer
to mapjoin, it will be easier to see the core of this change.  Thanks!


ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java
<https://reviews.apache.org/r/27247/#comment99831>

    Would it make sense to rename it as SparkMapJoinOptimizer to go more with existing convention?
 I understand the convention is not very well doc'ed (or even followed).



ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java
<https://reviews.apache.org/r/27247/#comment99832>

    Let's get rid of the commented code in this class (unless its necessary) to make it easier
to read.



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java
<https://reviews.apache.org/r/27247/#comment99833>

    We can revert this class, as these changes are not consumed by mapjoin.



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java
<https://reviews.apache.org/r/27247/#comment99834>

    We can revert this class as methods are not being used for mapjoin.



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java
<https://reviews.apache.org/r/27247/#comment99835>

    We need to put the check of auto.convert.join before we enable this optimization.  It
would make it easier this way.



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java
<https://reviews.apache.org/r/27247/#comment99836>

    I don't see this class attached



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java
<https://reviews.apache.org/r/27247/#comment99837>

    Let's revert this code, as its not being used for mapjoin.


- Szehon Ho


On Oct. 27, 2014, 9:53 p.m., Suhas Satish wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27247/
> -----------------------------------------------------------
> 
> (Updated Oct. 27, 2014, 9:53 p.m.)
> 
> 
> Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> This addresses jira HIVE-8616
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 46dcfaf 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java ed88c60

>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 8e28887 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java 4f5feca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 1c663c4 
> 
> Diff: https://reviews.apache.org/r/27247/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Suhas Satish
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message