pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy" <rohini.adi...@gmail.com>
Subject Re: Review Request 27169: [PIG-3979] - group all performance, garbage collection, and incremental aggregation
Date Sun, 02 Nov 2014 13:24:16 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27169/
-----------------------------------------------------------

(Updated Nov. 2, 2014, 1:24 p.m.)


Review request for pig.


Changes
-------

Removed sizereduction from spill(), but made size reduction kick off in the main thread by
changing the number of records to sample to the current record count. This is to avoid aggregation
happening in spill thread which might not have all thread local variables set as expected.


Bugs: PIG-3979
    https://issues.apache.org/jira/browse/PIG-3979


Repository: pig


Description
-------

Fixed a couple of issues with POPartialAgg
    - Made the spill of POPartialAgg synchronous so that System.gc() in SpillableMemoryManager
actually frees up memory.
    - Avoid lot of redundant aggregateSecondLevel() calls
    - Fixed the SpillableMemoryManager to not invoke extraGC if POPartialAgg
    - Made variables transient which are not required to be serialized in the plan


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartialAgg.java
1635881 
  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/GroupingSpillable.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/SpillableMemoryManager.java
1635881 
  http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestPOPartialAgg.java
1635881 

Diff: https://reviews.apache.org/r/27169/diff/


Testing
-------

Unit tests added to TestPOPartialAgg. Ran couple of manual e2e tests to check behaviour.


Thanks,

Rohini Palaniswamy


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message