hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pi Song (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-235) Performance issues with memory spills.
Date Thu, 15 May 2008 14:31:55 GMT

    [ https://issues.apache.org/jira/browse/PIG-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597144#action_12597144
] 

Pi Song commented on PIG-235:
-----------------------------

Amir,

If it works, there is another experimental option called "pig.spill.gc.activation.size" .
Could you please also try that out? If it is good, we may make it a default setting.

> Performance issues with memory spills.
> --------------------------------------
>
>                 Key: PIG-235
>                 URL: https://issues.apache.org/jira/browse/PIG-235
>             Project: Pig
>          Issue Type: Bug
>         Environment: Pig + Hadoop
>            Reporter: Amir Youssefi
>            Assignee: Utkarsh Srivastava
>            Priority: Critical
>
> We have are hitting low performance issue with memory spills.
> A reducer gets stuck in following state for *tens* of hours while
> thousands of small files are spilled. This is besides skewed keys issue.
> Note that size of spills become smaller as times goes. We can use this
> and try to address the issue by spilling in larger chunks.
> I tried different sub-set of data. Then made it work with all kind of tricks/hacks but
would like to have this working easily to say the least: 
> cogroup large_data_set by $0, small_date_set by $0
> small_date_set fits in memory.
> Log:
> 2008-05-06 23:24:06,014 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251352480(245461K) committed =
> 437256192(427008K) max = 477233152(466048K)
> 2008-05-06 23:24:06,734 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251401304(245509K) committed =
> 437256192(427008K) max = 477233152(466048K)
> 2008-05-06 23:24:07,455 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251254912(245366K) committed =
> 437256192(427008K) max = 477233152(466048K)
> 2008-05-06 23:24:08,175 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251281808(245392K) committed =
> 437256192(427008K) max = 477233152(466048K)
> 2008-05-06 23:24:08,895 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251309400(245419K) committed =
> 437256192(427008K) max = 477233152(466048K)
> 2008-05-06 23:24:09,615 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251358232(245467K) committed =
> 437256192(427008K) max = 477233152(466048K)
> 2008-05-06 23:24:10,336 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251267696(245378K) committed =
> 437256192(427008K) max = 477233152(466048K)
> 2008-05-06 23:24:11,056 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251307352(245417K) committed =
> 437256192(427008K) max = 477233152(466048K)
> 2008-05-06 23:24:11,776 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> called init = 4194304(4096K) used = 251335344(245444K) committed =
> 437256192(427008K) max = 477233152(466048K)
> (used column slowly increasing)
> Actual spill example:
>  8000 files 
> Sorting by time (new to old) I see small spills:
> -rw-------  1 amiry users     3675 May  6 23:44 pigbag635090.tmp
> -rw-------  1 amiry users        4 May  6 23:44 pigbag635091.tmp
> -rw-------  1 amiry users     3917 May  6 23:44 pigbag635086.tmp
> -rw-------  1 amiry users     3949 May  6 23:44 pigbag635088.tmp
> -rw-------  1 amiry users        4 May  6 23:44 pigbag635089.tmp
> -rw-------  1 amiry users     3969 May  6 23:44 pigbag635084.tmp
> -rw-------  1 amiry users     4073 May  6 23:44 pigbag635068.tmp
> -rw-------  1 amiry users    47634 May  6 23:44 pigbag635070.tmp
> -rw-------  1 amiry users     5101 May  6 23:44 pigbag635065.tmp
> -rw-------  1 amiry users     5722 May  6 23:44 pigbag635059.tmp
> -rw-------  1 amiry users     7570 May  6 23:44 pigbag635062.tmp
> -rw-------  1 amiry users     7802 May  6 23:44 pigbag635056.tmp
> -rw-------  1 amiry users     7514 May  6 23:44 pigbag635051.tmp
> -rw-------  1 amiry users     3929 May  6 23:44 pigbag635054.tmp
> -rw-------  1 amiry users     5342 May  6 23:44 pigbag635045.tmp
> -rw-------  1 amiry users     7361 May  6 23:44 pigbag635048.tmp
> -rw-------  1 amiry users     6663 May  6 23:44 pigbag635042.tmp
> -rw-------  1 amiry users     7511 May  6 23:44 pigbag635036.tmp
> -rw-------  1 amiry users     7520 May  6 23:44 pigbag635039.tmp
> -rw-------  1 amiry users     3873 May  6 23:44 pigbag635034.tmp
> -rw-------  1 amiry users     4029 May  6 23:44 pigbag635032.tmp
> -rw-------  1 amiry users     3823 May  6 23:44 pigbag635028.tmp
> -rw-------  1 amiry users     3726 May  6 23:44 pigbag635030.tmp
> -rw-------  1 amiry users     3934 May  6 23:44 pigbag635024.tmp
> Sorting by time (old to new) I see a few large spills then quickly (in
> less than 15min) come small ones:
> -rw-------  1 amiry users 45221453 May  6 21:23 pigbag59657.tmp
> -rw-------  1 amiry users 56161613 May  6 21:23 pigbag59658.tmp
> -rw-------  1 amiry users 70661942 May  6 21:23 pigbag59659.tmp
> -rw-------  1 amiry users 75308107 May  6 21:23 pigbag59660.tmp
> -rw-------  1 amiry users 76381091 May  6 21:24 pigbag59661.tmp
> -rw-------  1 amiry users 74691366 May  6 21:24 pigbag81914.tmp
> -rw-------  1 amiry users 73133098 May  6 21:24 pigbag103839.tmp
> -rw-------  1 amiry users 72750330 May  6 21:24 pigbag125123.tmp
> -rw-------  1 amiry users 71267460 May  6 21:25 pigbag146472.tmp
> -rw-------  1 amiry users 69638363 May  6 21:25 pigbag167358.tmp
> -rw-------  1 amiry users 68010250 May  6 21:25 pigbag187566.tmp
> -rw-------  1 amiry users 66312739 May  6 21:25 pigbag207447.tmp
> -rw-------  1 amiry users 64601422 May  6 21:26 pigbag226895.tmp
> -rw-------  1 amiry users 62997501 May  6 21:26 pigbag245690.tmp
> -rw-------  1 amiry users 62525926 May  6 21:26 pigbag264154.tmp
> -rw-------  1 amiry users 60940107 May  6 21:26 pigbag282367.tmp
> -rw-------  1 amiry users 59540198 May  6 21:26 pigbag300215.tmp
> -rw-------  1 amiry users 57918140 May  6 21:27 pigbag317750.tmp
> -rw-------  1 amiry users 57728845 May  6 21:27 pigbag334505.tmp
> -rw-------  1 amiry users 55427771 May  6 21:27 pigbag351436.tmp
> -rw-------  1 amiry users 55405942 May  6 21:27 pigbag367615.tmp
> -rw-------  1 amiry users 54600778 May  6 21:28 pigbag383872.tmp
> -rw-------  1 amiry users 52438311 May  6 21:28 pigbag399722.tmp
> -rw-------  1 amiry users 51250459 May  6 21:28 pigbag415094.tmp
> -rw-------  1 amiry users 50489324 May  6 21:28 pigbag430026.tmp
> -rw-------  1 amiry users 48311361 May  6 21:28 pigbag444835.tmp
> -rw-------  1 amiry users 47296555 May  6 21:28 pigbag458869.tmp
> -rw-------  1 amiry users 45703372 May  6 21:29 pigbag472771.tmp
> -rw-------  1 amiry users 46243949 May  6 21:29 pigbag486062.tmp
> -rw-------  1 amiry users 46195603 May  6 21:29 pigbag499549.tmp
> -rw-------  1 amiry users 43916731 May  6 21:29 pigbag513154.tmp
> -rw-------  1 amiry users 42970027 May  6 21:29 pigbag525921.tmp
> -rw-------  1 amiry users     5965 May  6 21:29 pigbag538555.tmp
> -rw-------  1 amiry users 46288099 May  6 21:30 pigbag538558.tmp
> -rw-------  1 amiry users     7735 May  6 21:30 pigbag539078.tmp
> -rw-------  1 amiry users     8058 May  6 21:30 pigbag539075.tmp
> -rw-------  1 amiry users 34034358 May  6 21:30 pigbag539079.tmp
> -rw-------  1 amiry users     8800 May  6 21:30 pigbag549021.tmp
> -rw-------  1 amiry users 45054789 May  6 21:31 pigbag549025.tmp
> -rw-------  1 amiry users     5750 May  6 21:31 pigbag549950.tmp
> -rw-------  1 amiry users     6794 May  6 21:31 pigbag549953.tmp
> -rw-------  1 amiry users 35112392 May  6 21:31 pigbag549956.tmp
> -rw-------  1 amiry users     7330 May  6 21:31 pigbag560146.tmp
> -rw-------  1 amiry users     7856 May  6 21:31 pigbag560143.tmp
> -rw-------  1 amiry users 44039882 May  6 21:32 pigbag560149.tmp
> -rw-------  1 amiry users     7508 May  6 21:32 pigbag561917.tmp
> -rw-------  1 amiry users     4567 May  6 21:32 pigbag561921.tmp
> -rw-------  1 amiry users 33031655 May  6 21:32 pigbag561924.tmp
> -rw-------  1 amiry users     7744 May  6 21:32 pigbag571510.tmp
> -rw-------  1 amiry users     7709 May  6 21:32 pigbag571507.tmp
> -rw-------  1 amiry users 42771860 May  6 21:33 pigbag571513.tmp
> -rw-------  1 amiry users 29912726 May  6 21:33 pigbag572784.tmp
> -rw-------  1 amiry users 42563305 May  6 21:34 pigbag581561.tmp
> -rw-------  1 amiry users 29961190 May  6 21:34 pigbag583024.tmp
> -rw-------  1 amiry users     7704 May  6 21:34 pigbag591783.tmp
> -rw-------  1 amiry users 41448656 May  6 21:35 pigbag591786.tmp
> -rw-------  1 amiry users     7679 May  6 21:35 pigbag592482.tmp
> -rw-------  1 amiry users     7328 May  6 21:35 pigbag592488.tmp
> -rw-------  1 amiry users     7809 May  6 21:35 pigbag592485.tmp
> -rw-------  1 amiry users   128909 May  6 21:35 pigbag592491.tmp
> -rw-------  1 amiry users 33069409 May  6 21:35 pigbag592530.tmp
> -rw-------  1 amiry users     7004 May  6 21:35 pigbag602114.tmp
> -rw-------  1 amiry users     4083 May  6 21:35 pigbag602121.tmp
> -rw-------  1 amiry users    11533 May  6 21:35 pigbag602117.tmp
> -rw-------  1 amiry users     7603 May  6 21:35 pigbag602124.tmp
> -rw-------  1 amiry users     7611 May  6 21:35 pigbag602130.tmp
> -rw-------  1 amiry users     7692 May  6 21:35 pigbag602127.tmp
> -rw-------  1 amiry users     7821 May  6 21:35 pigbag602136.tmp
> -rw-------  1 amiry users     7467 May  6 21:35 pigbag602133.tmp
> -rw-------  1 amiry users     1147 May  6 21:35 pigbag602139.tmp
> -rw-------  1 amiry users     8543 May  6 21:35 pigbag602144.tmp
> -rw-------  1 amiry users     8517 May  6 21:35 pigbag602141.tmp
> -rw-------  1 amiry users     3893 May  6 21:35 pigbag602150.tmp
> -rw-------  1 amiry users     6029 May  6 21:35 pigbag602147.tmp
> -rw-------  1 amiry users     7687 May  6 21:35 pigbag602152.tmp
> -rw-------  1 amiry users     7411 May  6 21:35 pigbag602158.tmp
> -rw-------  1 amiry users     6079 May  6 21:35 pigbag602155.tmp
> -rw-------  1 amiry users     6964 May  6 21:35 pigbag602161.tmp
> -rw-------  1 amiry users     5404 May  6 21:35 pigbag602168.tmp
> -rw-------  1 amiry users     7915 May  6 21:35 pigbag602164.tmp
> -rw-------  1 amiry users     4132 May  6 21:35 pigbag602174.tmp
> -rw-------  1 amiry users     7708 May  6 21:35 pigbag602171.tmp
> -rw-------  1 amiry users     7802 May  6 21:35 pigbag602176.tmp
> -rw-------  1 amiry users     6089 May  6 21:35 pigbag602182.tmp
> -rw-------  1 amiry users     8161 May  6 21:35 pigbag602179.tmp
> -rw-------  1 amiry users     7356 May  6 21:35 pigbag602188.tmp
> -rw-------  1 amiry users     8231 May  6 21:35 pigbag602185.tmp
> -rw-------  1 amiry users     8340 May  6 21:35 pigbag602191.tmp
> -rw-------  1 amiry users     3870 May  6 21:35 pigbag602198.tmp
> -rw-------  1 amiry users     7587 May  6 21:35 pigbag602195.tmp
> -rw-------  1 amiry users     5884 May  6 21:35 pigbag602200.tmp
> -rw-------  1 amiry users     7644 May  6 21:35 pigbag602206.tmp
> -rw-------  1 amiry users     7756 May  6 21:35 pigbag602203.tmp
> -rw-------  1 amiry users     8054 May  6 21:35 pigbag602209.tmp
> -rw-------  1 amiry users     6182 May  6 21:35 pigbag602215.tmp
> -rw-------  1 amiry users     7740 May  6 21:35 pigbag602212.tmp
> -rw-------  1 amiry users     6264 May  6 21:35 pigbag602222.tmp
> -rw-------  1 amiry users     7709 May  6 21:35 pigbag602218.tmp
> -rw-------  1 amiry users    10195 May  6 21:35 pigbag602225.tmp
> -rw-------  1 amiry users     7914 May  6 21:35 pigbag602232.tmp
> -rw-------  1 amiry users     4170 May  6 21:35 pigbag602229.tmp
> -rw-------  1 amiry users     7670 May  6 21:35 pigbag602235.tmp
> -rw-------  1 amiry users     2823 May  6 21:35 pigbag602241.tmp
> -rw-------  1 amiry users     6947 May  6 21:35 pigbag602238.tmp
> -rw-------  1 amiry users     7311 May  6 21:35 pigbag602246.tmp
> -rw-------  1 amiry users     7521 May  6 21:35 pigbag602243.tmp
> -rw-------  1 amiry users     7772 May  6 21:35 pigbag602249.tmp
> -rw-------  1 amiry users     5842 May  6 21:35 pigbag602255.tmp
> -rw-------  1 amiry users     7689 May  6 21:35 pigbag602252.tmp
> -rw-------  1 amiry users     5399 May  6 21:35 pigbag602261.tmp
> -rw-------  1 amiry users     5951 May  6 21:35 pigbag602258.tmp
> -rw-------  1 amiry users     7717 May  6 21:35 pigbag602264.tmp
> -rw-------  1 amiry users     4911 May  6 21:35 pigbag602271.tmp
> -rw-------  1 amiry users    11384 May  6 21:35 pigbag602267.tmp
> -rw-------  1 amiry users     6714 May  6 21:35 pigbag602277.tmp
> -rw-------  1 amiry users     6639 May  6 21:35 pigbag602274.tmp
> -rw-------  1 amiry users     7807 May  6 21:35 pigbag602280.tmp
> -rw-------  1 amiry users     6754 May  6 21:35 pigbag602286.tmp
> -rw-------  1 amiry users     6159 May  6 21:35 pigbag602283.tmp
> -rw-------  1 amiry users     7643 May  6 21:35 pigbag602292.tmp
> -rw-------  1 amiry users     7636 May  6 21:35 pigbag602289.tmp
> -rw-------  1 amiry users     6007 May  6 21:35 pigbag602295.tmp
> -rw-------  1 amiry users     7245 May  6 21:35 pigbag602301.tmp
> -rw-------  1 amiry users     7742 May  6 21:35 pigbag602298.tmp
> -rw-------  1 amiry users     1994 May  6 21:35 pigbag602304.tmp
> -rw-------  1 amiry users     8080 May  6 21:35 pigbag602309.tmp
> -rw-------  1 amiry users     8030 May  6 21:35 pigbag602306.tmp
> -rw-------  1 amiry users     6009 May  6 21:35 pigbag602315.tmp
> -rw-------  1 amiry users     5019 May  6 21:35 pigbag602312.tmp
> -rw-------  1 amiry users     3746 May  6 21:35 pigbag602318.tmp
> -rw-------  1 amiry users     5977 May  6 21:35 pigbag602323.tmp

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message