hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Jobtracker throws GC overhead limit exceeded
Date Tue, 05 Jul 2011 06:55:22 GMT
Meghana,

What is the heapsize for your JT? Try increasing that.

Also, we've fixed a huge number of issues in the JT (and Hadoop overall) since 0.19. Can you
upgrade to 0.20.203, the latest stable release?

thanks,
Arun

Sent from my iPhone

On Jul 4, 2011, at 11:10 PM, Meghana <meghana.marathe@germinait.com> wrote:

> Hi,
> 
> We are using Hadoop 0.19 on a small cluster of 7 machines (7 datanodes, 4
> task trackers), and we typically have 3-4 jobs running at a time. We have
> been facing the following error on the Jobtracker:
> 
> java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded
> 
> It seems to be thrown by RunningJob.killJob() and usually occurs after a day
> or so of starting up the cluster.
> 
> In the Jobtracker's output file:
> Exception in thread "initJobs" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
>    at java.lang.String.substring(String.java:1939)
>    at java.lang.String.substring(String.java:1904)
>    at org.apache.hadoop.fs.Path.getName(Path.java:188)
>    at
> org.apache.hadoop.fs.ChecksumFileSystem.isChecksumFile(ChecksumFileSystem.java:70)
>    at
> org.apache.hadoop.fs.ChecksumFileSystem$1.accept(ChecksumFileSystem.java:442)
>    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:726)
>    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:748)
>    at
> org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:457)
>    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:723)
>    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:748)
>    at
> org.apache.hadoop.mapred.JobHistory$JobInfo.getJobHistoryFileName(JobHistory.java:660)
>    at
> org.apache.hadoop.mapred.JobHistory$JobInfo.finalizeRecovery(JobHistory.java:746)
>    at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:1532)
>    at
> org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2232)
>    at
> org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:1938)
>    at
> org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:1953)
>    at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2012)
>    at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$JobInitThread.run(EagerTaskInitializationListener.java:62)
>    at java.lang.Thread.run(Thread.java:619)
> 
> 
> Please help!
> 
> Thanks,
> 
> Meghana

Mime
View raw message