hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bradford Stephens <bradfordsteph...@gmail.com>
Subject Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Date Sun, 26 Sep 2010 20:19:15 GMT
Hrm.... no. I've lowered it to -1, but I can try 1 again.

On Sun, Sep 26, 2010 at 6:47 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> Have you tried lowering mapred.job.reuse.jvm.num.tasks ?
>
> On Sun, Sep 26, 2010 at 3:30 AM, Bradford Stephens <
> bradfordstephens@gmail.com> wrote:
>
>> Nope, that didn't seem to help.
>>
>> On Sun, Sep 26, 2010 at 1:00 AM, Bradford Stephens
>> <bradfordstephens@gmail.com> wrote:
>> > I'm going to try running it on high-RAM boxes with -Xmx4096m or so,
>> > see if that helps.
>> >
>> > On Sun, Sep 26, 2010 at 12:55 AM, Bradford Stephens
>> > <bradfordstephens@gmail.com> wrote:
>> >> Greetings,
>> >>
>> >> I'm running into a brain-numbing problem on Elastic MapReduce. I'm
>> >> running a decent-size task (22,000 mappers, a ton of GZipped input
>> >> blocks, ~1TB of data) on 40 c1.xlarge nodes (7 gb RAM, ~8 "cores").
>> >>
>> >> I get failures randomly --- sometimes at the end of my 6-step process,
>> >> sometimes at the first reducer phase, sometimes in the mapper. It
>> >> seems to fail in multiple areas. Mostly in the reducers. Any ideas?
>> >>
>> >> Here's the settings I've changed:
>> >> -Xmx400m
>> >> 6 max mappers
>> >> 1 max reducer
>> >> 1GB swap partition
>> >> mapred.job.reuse.jvm.num.tasks=50
>> >> mapred.reduce.parallel.copies=3
>> >>
>> >>
>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
>> >>        at java.nio.CharBuffer.wrap(CharBuffer.java:350)
>> >>        at java.nio.CharBuffer.wrap(CharBuffer.java:373)
>> >>        at
>> java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
>> >>        at java.lang.StringCoding.decode(StringCoding.java:173)
>> >>        at java.lang.String.(String.java:443)
>> >>        at java.lang.String.(String.java:515)
>> >>        at
>> org.apache.hadoop.io.WritableUtils.readString(WritableUtils.java:116)
>> >>        at
>> cascading.tuple.TupleInputStream.readString(TupleInputStream.java:144)
>> >>        at
>> cascading.tuple.TupleInputStream.readType(TupleInputStream.java:154)
>> >>        at
>> cascading.tuple.TupleInputStream.getNextElement(TupleInputStream.java:101)
>> >>        at
>> cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:75)
>> >>        at
>> cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:33)
>> >>        at
>> cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:74)
>> >>        at
>> cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:34)
>> >>        at
>> cascading.tuple.hadoop.DeserializerComparator.compareTuples(DeserializerComparator.java:142)
>> >>        at
>> cascading.tuple.hadoop.GroupingSortingComparator.compare(GroupingSortingComparator.java:55)
>> >>        at
>> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
>> >>        at
>> org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:136)
>> >>        at
>> org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
>> >>        at
>> org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
>> >>        at
>> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
>> >>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
>> >>        at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2645)
>> >>        at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2586)
>> >>
>> >> --
>> >> Bradford Stephens,
>> >> Founder, Drawn to Scale
>> >> drawntoscalehq.com
>> >> 727.697.7528
>> >>
>> >> http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
>> >> solution. Process, store, query, search, and serve all your data.
>> >>
>> >> http://www.roadtofailure.com -- The Fringes of Scalability, Social
>> >> Media, and Computer Science
>> >>
>> >
>> >
>> >
>> > --
>> > Bradford Stephens,
>> > Founder, Drawn to Scale
>> > drawntoscalehq.com
>> > 727.697.7528
>> >
>> > http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
>> > solution. Process, store, query, search, and serve all your data.
>> >
>> > http://www.roadtofailure.com -- The Fringes of Scalability, Social
>> > Media, and Computer Science
>> >
>>
>>
>>
>> --
>> Bradford Stephens,
>> Founder, Drawn to Scale
>> drawntoscalehq.com
>> 727.697.7528
>>
>> http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
>> solution. Process, store, query, search, and serve all your data.
>>
>> http://www.roadtofailure.com -- The Fringes of Scalability, Social
>> Media, and Computer Science
>>
>



-- 
Bradford Stephens,
Founder, Drawn to Scale
drawntoscalehq.com
727.697.7528

http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
solution. Process, store, query, search, and serve all your data.

http://www.roadtofailure.com -- The Fringes of Scalability, Social
Media, and Computer Science

Mime
View raw message