Mailing-List: contact dev-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@pig.apache.org
Date: Thu, 14 Feb 2013 17:24:12 +0000 (UTC)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Message-ID: <JIRA.12629813.1359502036296.289474.1360862652754@arcas>
In-Reply-To: <JIRA.12629813.1359502036296@arcas>
References: <JIRA.12629813.1359502036296@arcas>
Subject: [jira] [Commented] (PIG-3148) OutOfMemory exception while spilling
 stale DefaultDataBag. Extra option to gc() before spilling large bag.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/PIG-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578507#comment-13578507 ] 

Koji Noguchi commented on PIG-3148:
-----------------------------------

Thanks Dmitriy, Rohini!  I like the fixed ratio suggestion.  Is 5% ok?  Maybe 10%?
Also, do we still want a configurable flag to enable this feature?
                
> OutOfMemory exception while spilling stale DefaultDataBag. Extra option to gc() before spilling large bag.
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-3148
>                 URL: https://issues.apache.org/jira/browse/PIG-3148
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>         Attachments: pig-3148-v01.patch
>
>
> Our user reported that one of their jobs in pig 0.10 occasionally failed with 
> 'Error: GC overhead limit exceeded' or 'Error: Java heap space', but rerunning it sometimes finishes successfully.
> For 1G heap reducer, heap dump showed it contained two huge DefaultDataBag with 300-400MBytes each when failing with OOM.
> Jstack at the time of OOM always showed that spill was running.
> {noformat}
> "Low Memory Detector" daemon prio=10 tid=0xb9c11800 nid=0xa52 runnable [0xb9afc000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.io.FileOutputStream.writeBytes(Native Method)
> 	at java.io.FileOutputStream.write(FileOutputStream.java:260)
> 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> 	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> 	- locked <0xe57c6390> (a java.io.BufferedOutputStream)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:90)
> 	- locked <0xe57c60b8> (a java.io.DataOutputStream)
> 	at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
> 	at org.apache.pig.data.utils.SedesHelper.writeBytes(SedesHelper.java:46)
> 	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:537)
> 	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
> 	at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
> 	at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
> 	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
> 	at org.apache.pig.data.DefaultDataBag.spill(DefaultDataBag.java:106)
> 	- locked <0xceb16190> (a java.util.ArrayList)
> 	at org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
> 	- locked <0xbeb86318> (a java.util.LinkedList)
> 	at sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
> 	at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
> 	at sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
> 	at sun.management.Sensor.trigger(Sensor.java:120)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira