pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3148) OutOfMemory exception while spilling stale DefaultDataBag. Extra option to gc() before spilling large bag.
Date Tue, 29 Jan 2013 23:49:12 GMT

    [ https://issues.apache.org/jira/browse/PIG-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565975#comment-13565975
] 

Koji Noguchi commented on PIG-3148:
-----------------------------------

I cannot attach the original query, but to give you an idea

{noformat}
A = LOAD '$INPUT' USING MyLoader('\u0001') AS
( val1, val2, val3, val4, val5, val6, val7, val8, val9, val10, val11, val12, val13, val14,
val15, val16);
B = GROUP A BY (val1, val2, val3, val8) PARALLEL $NUM_REDUCERS;
C = FOREACH B {
    D = FILTER A BY (val3 == 'status1' AND val5 == 'status2');
    E = D.val4;
    F = DISTINCT E;
    G = FILTER D BY val7 == 'status3';
GENERATE group.val1, group.val2, group.val8, COUNT(F), COUNT(G), SUM(G.val9), SUM(G.val10),
SUM(G.val11), SUM(A.val12), SUM(A.val13), SUM(A.val15), SUM(A.val14), SUM(A.val16);
}

STORE C INTO '$OUTPUT' USING PigStorage('\u0001');
{noformat}

Assuming that this script does not require a two huge DefaultDataBag, looked into SpillableMemoryManager.handleNotification.
  'handleNotification' is called whenever certain memory condition is met but not necessary
after the gc().  

What was happening on this user's case was, 
(i)  400MB of DefaultDataBag#1 goes stale.
(ii) SpillableMemoryManager.handleNotification is triggered.
(iii) Since gc() is not called yet, WeakReference is still valid and pig decides to spill
holding the lock of this DefaultDataBag#1.mContents(ArrayList).
(iv) While reduce task is working on another 400MB DefaultDataBag#2, jvm heap gets full and
gc is called.  Even though no one is using the stale DefaultDataBag#1, it cannot be GC-ed
since spill is holding the lock.

As a result, we end up with two DefaultDataBag leading to OOM.
                
> OutOfMemory exception while spilling stale DefaultDataBag. Extra option to gc() before
spilling large bag.
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-3148
>                 URL: https://issues.apache.org/jira/browse/PIG-3148
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>
> Our user reported that one of their jobs in pig 0.10 occasionally failed with 
> 'Error: GC overhead limit exceeded' or 'Error: Java heap space', but rerunning it sometimes
finishes successfully.
> For 1G heap reducer, heap dump showed it contained two huge DefaultDataBag with 300-400MBytes
each when failing with OOM.
> Jstack at the time of OOM always showed that spill was running.
> {noformat}
> "Low Memory Detector" daemon prio=10 tid=0xb9c11800 nid=0xa52 runnable [0xb9afc000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.io.FileOutputStream.writeBytes(Native Method)
> 	at java.io.FileOutputStream.write(FileOutputStream.java:260)
> 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> 	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> 	- locked <0xe57c6390> (a java.io.BufferedOutputStream)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:90)
> 	- locked <0xe57c60b8> (a java.io.DataOutputStream)
> 	at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
> 	at org.apache.pig.data.utils.SedesHelper.writeBytes(SedesHelper.java:46)
> 	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:537)
> 	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
> 	at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
> 	at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
> 	at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
> 	at org.apache.pig.data.DefaultDataBag.spill(DefaultDataBag.java:106)
> 	- locked <0xceb16190> (a java.util.ArrayList)
> 	at org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
> 	- locked <0xbeb86318> (a java.util.LinkedList)
> 	at sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
> 	at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
> 	at sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
> 	at sun.management.Sensor.trigger(Sensor.java:120)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message