hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-636) PERFORMANCE: Use lightweight bag implementations which do not register with SpillableMemoryManager with Combiner
Date Tue, 27 Jan 2009 21:55:02 GMT

     [ https://issues.apache.org/jira/browse/PIG-636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Pradeep Kamath updated PIG-636:

    Attachment: PIG-636-v2.patch

Attached new version of patch with the following two changes as per review comments:
1) SingleTupleBag now only has one constructor which takes the tuple the bag is meant to contain.
This way SingleTupleBags can only be created with the member Tuple.
2) size() now returns 1.

> PERFORMANCE: Use lightweight bag implementations which do not register with SpillableMemoryManager
with Combiner
> ----------------------------------------------------------------------------------------------------------------
>                 Key: PIG-636
>                 URL: https://issues.apache.org/jira/browse/PIG-636
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: types_branch
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>             Fix For: types_branch
>         Attachments: PIG-636-v2.patch, PIG-636.patch
> Currently whenever Combiner is used in pig, in the map, the POPrecombinerLocalRearrange
operator puts the single "value" tuple corresponding to a key into a DataBag and passes this
to the foreach which is being combined. This will generate as many bags as there are input
records. These bags all will have a single tuple and hence are small and should not need to
be spilt to disk. However since the bags are created through the BagFactory mechanism, each
bag creation is registered with the SpillableMemoryManager and a weak reference to the bag
is stored in a linked list. This linked list grows really big over time causing unnecessary
Garbage collection runs. This can be avoided by having a simple lightweight implementation
of the DataBag interface to store the single tuple in a bag. Also these SingleTupleBags should
be created without registering with the spillableMemoryManager. Likewise the bags created
in POCombinePackage are supposed to fit in Memory and not spill. Again a NonSpillableDataBag
implementation of DataBag interface which does not register with the SpillableMemoryManager
would help.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message