hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1209) Port POJoinPackage to proactively spill
Date Fri, 29 Jan 2010 21:53:34 GMT

    [ https://issues.apache.org/jira/browse/PIG-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806525#action_12806525

Pradeep Kamath commented on PIG-1209:

For testing, the data for the first input of the join can be a large number of records (so
that size in memory > 500 MB or so) with the same join key. This will hopefully spill and
fail with old code and not fail with new code.

> Port POJoinPackage to proactively spill
> ---------------------------------------
>                 Key: PIG-1209
>                 URL: https://issues.apache.org/jira/browse/PIG-1209
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Sriranjan Manjunath
>            Assignee: Ashutosh Chauhan
>             Fix For: 0.7.0
> POPackage proactively spills the bag whereas POJoinPackage still uses the SpillableMemoryManager.
We should port this to use InternalCacheBag which proactively spills.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message