pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (PIG-5384) OOM while spilling large bag
Date Tue, 19 Mar 2019 19:51:00 GMT
Koji Noguchi created PIG-5384:
---------------------------------

             Summary: OOM while spilling large bag 
                 Key: PIG-5384
                 URL: https://issues.apache.org/jira/browse/PIG-5384
             Project: Pig
          Issue Type: Improvement
            Reporter: Koji Noguchi
            Assignee: Koji Noguchi


One of the common OOM issue in Pig is, Pig hitting OOM while trying to spill a large bag.
Current solutions is to give higher heapsize or tweak 
{noformat}
pig.spill.memory.usage.threshold.fraction
pig.spill.collection.threshold.fraction
pig.spill.unused.memory.threshold.size
{noformat}
and make sure spilling starts early enough.  These params are still critical but wondering
if any improvement can be made to increase the chances of avoiding OOM while spilling a single
large bag.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message