hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho" <sze...@cloudera.com>
Subject Re: Review Request 27640: HIVE-8700 Replace ReduceSink to HashTableSink (or equi.) for small tables [Spark Branch]
Date Wed, 05 Nov 2014 21:23:36 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27640/#review60031
-----------------------------------------------------------


Hi, I know this is a combination of patches from Suhas and Chao, but I left all some comments/questions
here for the overall patch.


ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/27640/#comment101369>

    Please use proper javadoc notation for your javadocs.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/27640/#comment101365>

    Can you elaborate why we need this assumption?  This may not be true in all cases.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/27640/#comment101373>

    We need to handle the case of where parent is the mapwork of big-table.  Basically if
we are walking from big-table to a reduce work of mapjoin, we should not replace RS with HTS.
 (it can happen for example, if big-table is the result from a group by).
    
    Only work of small-tables should get replaced.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/27640/#comment101371>

    Let's just get mapJoinOp and check if its null in one call.  (Can get rid of one of the
methods).



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/27640/#comment101363>

    Fix indent here.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/27640/#comment101366>

    Please fix all these tabs, and make sure to indent them properly.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/27640/#comment101368>

    Are you sure we dont need to initialize the HTSOperator's values like it does in LocalMapJoinProcFactory?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/27640/#comment101367>

    Nit: 'parentOps' is probably a better name.


- Szehon Ho


On Nov. 5, 2014, 8:29 p.m., Suhas Satish wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27640/
> -----------------------------------------------------------
> 
> (Updated Nov. 5, 2014, 8:29 p.m.)
> 
> 
> Review request for hive, Chao Sun, Jimmy Xiang, Szehon Ho, and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> This replaces ReduceSinks with HashTableSinks in smaller tables for a map-join. But the
condition check field to detect map-join is actually being set in CommonJoinResolver, which
doesnt exist yet. We need to decide where is the right place to populate this field. 
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 795a5d7 
> 
> Diff: https://reviews.apache.org/r/27640/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Suhas Satish
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message