hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pi Song (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-170) Memory manager spills bags in the wrong order
Date Fri, 28 Mar 2008 13:44:24 GMT

    [ https://issues.apache.org/jira/browse/PIG-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583025#action_12583025

Pi Song commented on PIG-170:

In certain scenarios, for example, a lot of small intermediate bags are used to calculate
something and outputs are stored in a big bag. The big bag will become the first node after
Collections.sort(spillables, new Comparator<WeakReference<Spillable>>()

 and the below code will no longer work.
    public void registerSpillable(Spillable s) {
        synchronized(spillables) {
            // Cleaing the entire list is too expensive.  Just trim off the front while
            // we can.
            WeakReference<Spillable> first = spillables.peek();
            while (first != null && first.get() == null) {
                first = spillables.peek();
            spillables.add(new WeakReference<Spillable>(s));

> Memory manager spills bags in the wrong order
> ---------------------------------------------
>                 Key: PIG-170
>                 URL: https://issues.apache.org/jira/browse/PIG-170
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Olga Natkovich
>            Assignee: Amir Youssefi
>         Attachments: PIG-170_0_20080327.patch
> For optimal performance, we want to spill the largest bags first. This is not what is
happening right now and could be causing some of our memory issues.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message