pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy" <rohini.adi...@gmail.com>
Subject Re: Review Request 27169: [PIG-3979] - group all performance, garbage collection, and incremental aggregation
Date Fri, 24 Oct 2014 21:59:09 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27169/
-----------------------------------------------------------

(Updated Oct. 24, 2014, 9:59 p.m.)


Review request for pig.


Bugs: PIG-3979
    https://issues.apache.org/jira/browse/PIG-3979


Repository: pig


Description (updated)
-------

Fixed a couple of issues with POPartialAgg
    - Made the spill of POPartialAgg synchronous so that System.gc() in SpillableMemoryManager
actually frees up memory.
    - Avoid lot of redundant aggregateSecondLevel() calls
    - Fixed the SpillableMemoryManager to not invoke extraGC if POPartialAgg
    - Made variables transient which are not required to be serialized in the plan


Diffs
-----

  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartialAgg.java
1633956 
  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/SpillableMemoryManager.java
1633956 
  http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestPOPartialAgg.java
1633956 

Diff: https://reviews.apache.org/r/27169/diff/


Testing
-------

Unit tests added to TestPOPartialAgg. Ran couple of manual e2e tests to check behaviour.


Thanks,

Rohini Palaniswamy


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message