pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kelly zhang" <liyun.zh...@intel.com>
Subject Re: Review Request 32031: PIG-4193: Make collected group work with Spark
Date Fri, 13 Mar 2015 08:01:15 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32031/#review76344
-----------------------------------------------------------



src/org/apache/pig/backend/hadoop/executionengine/spark/operator/POCollectedGroupSpark.java
<https://reviews.apache.org/r/32031/#comment123908>

    add license text in the head


- kelly zhang


On March 13, 2015, 7:51 a.m., Praveen R wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32031/
> -----------------------------------------------------------
> 
> (Updated March 13, 2015, 7:51 a.m.)
> 
> 
> Review request for pig, liyun zhang and Mohit Sabharwal.
> 
> 
> Bugs: PIG-4193
>     https://issues.apache.org/jira/browse/PIG-4193
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Moved getNextTuple(boolean proceed) method from POCollectedGroup to POCollectedGroupSpark.
> 
> Collected group when used with mr performs group operation in the mapside after making
sure all data for same key exists on single map. This behaviour in spark is achieved by a
single map on function using POCollectedGroup operator.
> 
> TODO:
> - Avoid using rdd.count() in CollectedGroupConverter.
> 
> 
> Diffs
> -----
> 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCollectedGroup.java
7f2f18e52e083b3e8e90ba02d07f12bcbc9be859 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java ca7a45f33320064e22628b40b34be7b9f7b07c36

>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
3d04ba11855c39960e00d6f51b66654d1c70ebad 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/operator/POCollectedGroupSpark.java
PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/plan/SparkCompiler.java PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/32031/diff/
> 
> 
> Testing
> -------
> 
> Tested TestCollectedGroup and do not have any new successes or failures.
> 
> 
> Thanks,
> 
> Praveen R
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message