hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris K Wensel <ch...@wensel.net>
Subject Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Date Mon, 27 Sep 2010 00:09:31 GMT
Try using a lower threshold value (the num of values in the LRU to cache). this is the tradeoff
of this approach.

ckw

On Sep 26, 2010, at 4:46 PM, Bradford Stephens wrote:

> Sadly, making Chris's changes didn't help.
> 
> Here's the Cascading code, it's pretty simple but uses the new
> "combiner"-like functionality:
> 
> http://pastebin.com/ccvDmLSX
> 
> 
> 
> On Sun, Sep 26, 2010 at 9:37 AM, Ted Dunning <ted.dunning@gmail.com> wrote:
>> My feeling is that you have some kind of leak going on in your mappers or
>> reducers and that reducing the number of times the jvm is re-used would
>> improve matters.
>> 
>> GC overhead limit indicates that your (tiny) heap is full and collection is
>> not reducing that.
>> 
>> On Sun, Sep 26, 2010 at 12:55 AM, Bradford Stephens <
>> bradfordstephens@gmail.com> wrote:
>> 
>>> mapred.job.reuse.jvm.num.tasks=50
>>> 
>> 
> 
> 
> 
> -- 
> Bradford Stephens,
> Founder, Drawn to Scale
> drawntoscalehq.com
> 727.697.7528
> 
> http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
> solution. Process, store, query, search, and serve all your data.
> 
> http://www.roadtofailure.com -- The Fringes of Scalability, Social
> Media, and Computer Science
> 
> -- 
> You received this message because you are subscribed to the Google Groups "cascading-user"
group.
> To post to this group, send email to cascading-user@googlegroups.com.
> To unsubscribe from this group, send email to cascading-user+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
> 

--
Chris K Wensel
chris@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support, and licensing for Cascading


Mime
View raw message