pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1348) PigStorage making unnecessary byte array copy when storing data
Date Wed, 07 Apr 2010 20:56:33 GMT

    [ https://issues.apache.org/jira/browse/PIG-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854707#action_12854707
] 

Dmitriy V. Ryaboy commented on PIG-1348:
----------------------------------------

Umm I am totally wrong about the new object thing.

{code}
public static String valueOf(int i) {
           return Integer.toString(i, 10);
}
{code}

mea culpa.

Though it's still true that this approach can clean up / shorten the code, it's totally untrue
that there's some magical object reuse happening if you do it.

-D



> PigStorage making unnecessary byte array copy when storing data
> ---------------------------------------------------------------
>
>                 Key: PIG-1348
>                 URL: https://issues.apache.org/jira/browse/PIG-1348
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.7.0
>            Reporter: Ashutosh Chauhan
>            Assignee: Richard Ding
>             Fix For: 0.7.0
>
>         Attachments: PIG-1348.patch, PIG-1348_2.patch
>
>
> InternalCachedBag makes estimate of memory available to the VM by using Runtime.getRuntime().maxMemory().
It then uses 10%(by default, though configurable) of this memory and divides this memory into
number of bags. It keeps track of the memory used by bags and then proactively spills if bags
memory usage reach close to these limits. Given all this in theory when presented with data
more then it can handle InternalCachedBag should not run out of memory. But in practice we
find OOM happening. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message