pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy" <rohini.adi...@gmail.com>
Subject Re: Review Request 25617: PIG-4104: Accumulator UDF throws OOM in Tez
Date Sun, 14 Sep 2014 05:24:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25617/#review53276
-----------------------------------------------------------



http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/25617/#comment92868>

    Daniel,
       Would there be a problem if one instance of TezAccumulativeTupleBuffer was used for
each record as the ArrayList bags can be cleared and min key reset? I am only concerned with
the case of streaming. I am still not familiar with internals of streaming and I believe there
were cases copies of data had to be made for streaming.


- Rohini Palaniswamy


On Sept. 14, 2014, 5:20 a.m., Rohini Palaniswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25617/
> -----------------------------------------------------------
> 
> (Updated Sept. 14, 2014, 5:20 a.m.)
> 
> 
> Review request for pig, Cheolsoo Park and Daniel Dai.
> 
> 
> Bugs: PIG-4104
>     https://issues.apache.org/jira/browse/PIG-4104
> 
> 
> Repository: pig
> 
> 
> Description
> -------
> 
> Use a separate TezAccumulativeTupleBuffer that iterates through the inputs and returns
tuples in batches instead of making a full copy.
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/PigConfiguration.java
1624398 
>   http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java
1624398 
>   http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java
1624398 
>   http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
1624398 
>   http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/util/AccumulatorOptimizerUtil.java
1624398 
> 
> Diff: https://reviews.apache.org/r/25617/diff/
> 
> 
> Testing
> -------
> 
> Ran TestAccumulator in unit test and Accumulator, SecondarySort test groups in e2e and
they all passed. Will run the full suite before committing.
> 
> 
> Thanks,
> 
> Rohini Palaniswamy
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message