hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Can not start task tracker because java.lang.NullPointerException
Date Fri, 22 May 2009 20:53:12 GMT
Hi Lance,

Is it possible that your mapred.local.dir is in /tmp and you have a cron job
that cleans it up at night (default on many systems)?

Thanks
-Todd

On Fri, May 22, 2009 at 9:33 AM, Lance Riedel <lance@dotspots.com> wrote:

> Version 19.1 with patches:
> 4780-2v19.patch (Jira  4780)
> closeAll3.patch (Jira 3998)
> I have confirmed that
> https://issues.apache.org/jira/browse/HADOOP-4924patch is in, so that
> is not the fix.
>
>
> We are having task trackers die every night with a null pointer exception.
> Usually 2 or so out of 8 (25% each night).
>
>
> Here are the logs:
>
> Version 19.1 with
> 2009-05-22 02:46:49,911 INFO org.apache.hadoop.mapred.TaskTracker: Received
> 'KillJobAction' for job: job_200905211749_0451
> 2009-05-22 02:46:49,911 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_200905211749_0451_m_000000_0 done; removing files.
> 2009-05-22 02:46:54,911 INFO org.apache.hadoop.mapred.TaskTracker:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>
> taskTracker/jobcache/job_200905211749_0421/attempt_200905211749_0421_r_000009_0/output/file.out
> in any of the configured local directories
> 2009-05-22 02:47:13,968 INFO org.apache.hadoop.mapred.TaskTracker: Received
> 'KillJobAction' for job: job_200905211749_0444
> 2009-05-22 02:47:13,969 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_200905211749_0444_m_000000_0 done; removing files.
> 2009-05-22 02:47:18,968 INFO org.apache.hadoop.mapred.TaskTracker:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>
> taskTracker/jobcache/job_200905211749_0421/attempt_200905211749_0421_r_000009_0/output/file.out
> in any of the configured local directories
> 2009-05-22 02:48:52,324 INFO org.apache.hadoop.mapred.TaskTracker:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>
> taskTracker/jobcache/job_200905211749_0421/attempt_200905211749_0421_r_000009_0/output/file.out
> in any of the configured local directories
> 2009-05-22 02:49:10,779 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200905211749_0452_m_000006_0
> task's
> state:UNASSIGNED
> 2009-05-22 02:49:10,779 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to
> launch : attempt_200905211749_0452_m_000006_0
> 2009-05-22 02:49:10,779 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 4 and trying to launch
> attempt_200905211749_0452_m_000006_0
> 2009-05-22 02:49:15,274 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner
> jvm_200905211749_0452_m_1998728288 spawned.
> 2009-05-22 02:49:15,765 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_200905211749_0452_m_1998728288 given task:
> attempt_200905211749_0452_m_000006_0
> 2009-05-22 02:49:15,781 INFO org.apache.hadoop.mapred.TaskTracker:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>
> taskTracker/jobcache/job_200905211749_0421/attempt_200905211749_0421_r_000009_0/output/file.out
> in any of the configured local directories
> 2009-05-22 02:49:15,781 INFO org.apache.hadoop.mapred.TaskTracker:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>
> taskTracker/jobcache/job_200905211749_0452/attempt_200905211749_0452_m_000006_0/output/file.out
> in any of the configured local directories
> 2009-05-22 02:49:19,784 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_200905211749_0452_m_000006_0 1.0% hdfs://
>
> ec2-75-101-247-52.compute-1.amazonaws.com:54310/paragraphInstances/2009-05-22/rollup#06h#3e04c188-245a-4856-9a54-2fec60e85e3d.seq:0+9674259
> <
> http://ec2-75-101-247-52.compute-1.amazonaws.com:54310/paragraphInstances/2009-05-22/rollup#06h%233e04c188-245a-4856-9a54-2fec60e85e3d.seq:0+9674259
> >
> 2009-05-22 02:49:19,785 INFO org.apache.hadoop.mapred.TaskTracker: Task
> attempt_200905211749_0452_m_000006_0 is done.
> 2009-05-22 02:49:19,785 INFO org.apache.hadoop.mapred.TaskTracker: reported
> output size for attempt_200905211749_0452_m_000006_0  was 0
> 2009-05-22 02:49:19,787 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 4
> 2009-05-22 02:49:19,954 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200905211749_0452_m_1998728288 exited. Number of tasks it ran: 1
> 2009-05-22 02:59:19,297 INFO org.apache.hadoop.mapred.TaskTracker: Recieved
> RenitTrackerAction from JobTracker
> 2009-05-22 02:59:19,298 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because java.lang.NullPointerException
>        at
>
> org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:2300)
>        at
>
> org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:2273)
>        at org.apache.hadoop.mapred.TaskTracker.close(TaskTracker.java:840)
>        at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1728)
>        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2785)
>
> 2009-05-22 02:59:19,300 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at domU-12-31-38-01-AD-91/
> 10.253.178.95
> ************************************************************/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message