pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park" <piaozhe...@gmail.com>
Subject Review Request 15931: PIG-3585 Implement union in Tez
Date Sun, 01 Dec 2013 07:00:52 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15931/
-----------------------------------------------------------

Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.


Bugs: PIG-3585
    https://issues.apache.org/jira/browse/PIG-3585


Repository: pig-git


Description
-------

This patch implements union as follows: load vertices -> broadcast edges -> union vertex.

Th changes include:
* In the front-end, TezCompiler converts POUnion into a new vertex and connects it to its
predecessors with broadcast edges.
* In the back-end, a new POPackage class called POBroadcastTezLoad is added. This classes
implements TezLoad interface, and it pulls every record from ShuffledUnorderedKVInputs in
order and unions them.


Diffs
-----

  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java
e49de40 
  src/org/apache/pig/backend/hadoop/executionengine/tez/POBroadcastTezLoad.java e69de29 
  src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 9a2b499 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 529bf30 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java e3f5a5d 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java dcd6a5a 
  test/e2e/pig/tests/tez.conf 7fd5fb1 

Diff: https://reviews.apache.org/r/15931/diff/


Testing
-------

* New e2e test case is added.
* ant test-tez passes.
* All e2e tests pass.


Thanks,

Cheolsoo Park


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message