hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1037) better memory layout and spill for sorted and distinct bags
Date Mon, 02 Nov 2009 03:00:59 GMT

    [ https://issues.apache.org/jira/browse/PIG-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772410#action_12772410
] 

Ashutosh Chauhan commented on PIG-1037:
---------------------------------------

I am kinda late on this, but I would appreciate if someone can provide brief description of
how this patch improves the memory layout and alleviates the spill problem. I took a quick
look at the patch. 
According to my understanding, previously when memory is about to get exhausted Pig will start
writing to the disk one tuple at a time. With this new patch, once the memory limit is hit
whole bag is spilled to disk, at that point in-memory bag contains no tuples. If in-memory
bag fills again, all of its content are spilled to disk in entirety again and so on.. So this
patch ensures that we are not spilling one tuple at a time, but a full bag a time. Is this
correct or am I missing something ?

> better memory layout and spill for sorted and distinct bags
> -----------------------------------------------------------
>
>                 Key: PIG-1037
>                 URL: https://issues.apache.org/jira/browse/PIG-1037
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Olga Natkovich
>            Assignee: Ying He
>             Fix For: 0.6.0
>
>         Attachments: PIG-1037.patch, PIG-1037.patch2, PIG-1037.patch3
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message