pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai" <dai...@gmail.com>
Subject Re: Review Request 17529: [PIG-3732] Use ONE_TO_ONE edge and IdentityInOut in orderby intermediate vertex
Date Thu, 30 Jan 2014 22:24:11 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17529/#review33264
-----------------------------------------------------------

Ship it!


+1 for the rest

- Daniel Dai


On Jan. 30, 2014, 8:28 a.m., Rohini Palaniswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17529/
> -----------------------------------------------------------
> 
> (Updated Jan. 30, 2014, 8:28 a.m.)
> 
> 
> Review request for pig, Cheolsoo Park and Daniel Dai.
> 
> 
> Bugs: PIG-3732
>     https://issues.apache.org/jira/browse/PIG-3732
> 
> 
> Repository: pig
> 
> 
> Description
> -------
> 
> Orderby has 4 vertices and changes done are as below.
> 
> Load Vertex -> Partitioner Vertex 
>      - Was RoundRobinPartitioner with sorted shuffle and parallelism of Partitioner Vertex
was same as reducer vertex (i.e PARALLEL clause). Now ONE_TO_ONE unsorted edge between Load
Vertex and Partitioner Vertex with Partitioner Vertex having same parallelism as Load Vertex.
Will get the performance numbers for both cases by Friday.
> Load Vertex -> Sampler Vertex  
> Sampler Vertex -> Partitioner Vertex (Broadcast edge)
>      - The POPackage->POForeach->POLocalRearrange in Partitioner Vertex has been
replaced by POIdentityInOutTez
> Partitioner Vertex -> Reducer Vertex
> 
> Need to attempt this for Skewed Join as well.
> 
> 
> This patch also sets credential on DAG which is required after TEZ-395
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java
PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POLocalRearrangeTez.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDAG.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/NullablePartitionWritable.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/PigNullableWritable.java
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC16.gld
1562426 
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld
1562426 
> 
> Diff: https://reviews.apache.org/r/17529/diff/
> 
> 
> Testing
> -------
> 
> test-tez and tez.conf e2e tests pass
> 
> 
> Thanks,
> 
> Rohini Palaniswamy
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message