pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2829) Use partial aggregation more aggresively
Date Wed, 05 Sep 2012 07:02:08 GMT

    [ https://issues.apache.org/jira/browse/PIG-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448528#comment-13448528
] 

Dmitriy V. Ryaboy commented on PIG-2829:
----------------------------------------

Jie, sorry I missed this ticket before. As you may have seen, I completely reimplemented this
whole chunk of code in PIG-2888. Can you rerun your benchmarks and see if some of the improvements
you propose here should be applied to the code developed in that ticket?
                
> Use partial aggregation more aggresively
> ----------------------------------------
>
>                 Key: PIG-2829
>                 URL: https://issues.apache.org/jira/browse/PIG-2829
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: 0.10.0
>            Reporter: Jie Li
>         Attachments: 2829.1.patch, 2829.2.patch, 2829.separate.options.patch, pigmix-10G.png,
tpch-10G.png
>
>
> Partial aggregation (Hash Aggregation, aka in-map combiner) is a new feature in Pig 0.10
that will perform aggregation within map function. The main advantage against combiner is
it avoids de/serializing and sorting the data, and it can auto disable itself if the data
reduction rate is low. Currently it's disabled by default.
> To leverage the power of PartialAgg more aggressively, several things need to be revisited:
> 1. The threshold of auto-disabling. Currently each mapper looks at first 1k (hard-coded)
records to see if there's enough data size reduction (defaults to 10x, configurable). The
check would happen earlier if the hash table gets full before processing the 1k records (hash
table size is controlled by pig.cachedbag.memusage). We might want to relax these thresholds.
> 2. Dependency on the combiner. Currently the PartialAgg won't work without a combiner
following it, so we need to provide separate options to enable each independently. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message