pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy" <rohini.adi...@gmail.com>
Subject Review Request 25617: PIG-4104: Accumulator UDF throws OOM in Tez
Date Sun, 14 Sep 2014 05:20:38 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25617/
-----------------------------------------------------------

Review request for pig, Cheolsoo Park and Daniel Dai.


Bugs: PIG-4104
    https://issues.apache.org/jira/browse/PIG-4104


Repository: pig


Description
-------

Use a separate TezAccumulativeTupleBuffer that iterates through the inputs and returns tuples
in batches instead of making a full copy.


Diffs
-----

  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/PigConfiguration.java 1624398

  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java
1624398 
  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java
1624398 
  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
1624398 
  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/util/AccumulatorOptimizerUtil.java
1624398 

Diff: https://reviews.apache.org/r/25617/diff/


Testing
-------

Ran TestAccumulator in unit test and Accumulator, SecondarySort test groups in e2e and they
all passed. Will run the full suite before committing.


Thanks,

Rohini Palaniswamy


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message