hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Killed : GC overhead limit exceeded
Date Sat, 17 Jul 2010 05:28:27 GMT
Have you tried increasing memory beyond 1GB for your map task ?

I think you have noticed that both OOME came from Pattern.compile().

Please take a look at
http://www.docjar.com/html/api/java/lang/String.java.html

I would suggest pre-compiling the three patterns when setting up your mapper
- basically write your own split() and replaceAll().

I recently did something similar. You can find out the performance
improvement by customization -
https://issues.apache.org/jira/browse/MAPREDUCE-1946

Cheers

On Fri, Jul 16, 2010 at 6:06 AM, Some Body <somebody@squareplanet.de> wrote:

> Guess attachments are stripped.
>
> Here's the memory graph:   http://tinyurl.com/37g3hmu
> Here's the VM Summary:   http://tinyurl.com/36wqzjq
>
> Alan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message