hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Xiang" <jxi...@cloudera.com>
Subject Re: Review Request 28727: HIVE-8638 Implement bucket map join optimization [Spark Branch]
Date Sun, 07 Dec 2014 05:20:53 GMT


> On Dec. 5, 2014, 10:40 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java,
line 121
> > <https://reviews.apache.org/r/28727/diff/1/?file=782892#file782892line121>
> >
> >     Be aware of different hadoop versions.

I checked. Both hadoop 1 and 2 have this API.


> On Dec. 5, 2014, 10:40 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java,
line 321
> > <https://reviews.apache.org/r/28727/diff/1/?file=782895#file782895line321>
> >
> >     Can we have some comments on what this methods do, as it's not trivial?

This is changed, so it is simple now.


> On Dec. 5, 2014, 10:40 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java,
line 342
> > <https://reviews.apache.org/r/28727/diff/1/?file=782895#file782895line342>
> >
> >     Do we assume the set always contains only entry?

That's a good question. Let me fix this in the next patch.


> On Dec. 5, 2014, 10:40 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java,
line 96
> > <https://reviews.apache.org/r/28727/diff/1/?file=782893#file782893line96>
> >
> >     Are we only get the first instance of such operator? There could be more than
one instance in a work.

Yes, we get the first instance only. If there could be many instances, we need to fix it.


On Dec. 5, 2014, 10:40 p.m., Jimmy Xiang wrote:
> > Overall, the patch looks good to me. Besides the comments above, I'm not very confident
that I understand enough to let this go in. Maybe a face to face discussion will help.

Sure. Let's have a chat. Thanks.


- Jimmy


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28727/#review64093
-----------------------------------------------------------


On Dec. 4, 2014, 11:38 p.m., Jimmy Xiang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28727/
> -----------------------------------------------------------
> 
> (Updated Dec. 4, 2014, 11:38 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8638
>     https://issues.apache.org/jira/browse/HIVE-8638
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Patch v3 that works when bucket number matches
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 09c667e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SparkHashTableSinkOperator.java cfc1501

>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java
2f9e55a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
4054173 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkBucketJoinProcCtx.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java 8b78123

>   ql/src/test/queries/clientpositive/bucket_map_join_spark1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/bucket_map_join_spark2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_spark1.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/spark/bucket_map_join_spark2.q.out PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/28727/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message