hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <jonathan.hw...@accenture.com>
Subject Debug hadoop error
Date Tue, 17 May 2011 15:13:45 GMT
I need some help on figuring out why my job failed. I built a single node cluster just to try
it out. I follow the example link http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

Everything seems to be working correctly. I formated the namenode. Able to connect to all
my jobtracker, datanode, namenode via the WebUI. I was able to start and stop all the hadoop
services.

However, when I try to run the wordcount example, I got this: Error initializing attempt_201105161023_0002_m_000011_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135) at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken
does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129) ... 5
more

I created the directory on local file system. $ sudo mkdir /app/hadoop/tmp $ sudo chown hadoop:hadoop
/app/hadoop/tmp

Also modified in file conf/core-site.xml:

hadoop.tmp.dir /app/hadoop/tmp A base for other temporary directories.

When I format the namenode, it created the subdirectory on both local and HDFS successful.

When I look at the result of the wordcount faile ouput, the error message is complaints about
IO error, on file /app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/ job_201105161023_0002/jobToken

Did some troubleshooting, I can browse to this jobToken file on the local file system no problem.
content is something like HDTS MapReduce.job 201105161023_0002

So is it permission issue? I made owner of hadoop process able to write to all the subfolder
and it was able to create the file. So what else can be wrong?

What is that error message mean... so puzzling... wish there are better error messaging...

BELOW IS THE DETAIL OUTPUT FROM COMMAND LINE:


--------------------------------------------------------------------------------

hadoop@jonathan-VirtualBox:/usr/local/hadoop/hadoop-0.20.203.0$ bin/hadoop jar hadoop-examples-0.20.203.0.jar
wordcount app/download app/output4 11/05/16 13:38:56 INFO input.FileInputFormat: Total input
paths to process : 3 11/05/16 13:39:05 INFO mapred.JobClient: Running job: job_201105161222_0003
11/05/16 13:39:06 INFO mapred.JobClient: map 0% reduce 0% 11/05/16 13:39:17 INFO mapred.JobClient:
Task Id : attempt_201105161222_0003_m_000004_0, Status : FAILED Error initializing attempt_201105161222_0003_m_000004_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135) at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129) ... 5
more

11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stdout
11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stderr
11/05/16 13:39:22 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_1, Status
: FAILED Error initializing attempt_201105161222_0003_m_000004_1: java.io.IOException: Exception
reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135) at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129) ... 5
more

11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stdout
11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stderr
11/05/16 13:39:23 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_2, Status
: FAILED Error initializing attempt_201105161222_0003_m_000004_2: java.io.IOException: Exception
reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135) at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129) ... 5
more

11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stdout
11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stderr
11/05/16 13:39:30 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_0, Status
: FAILED Error initializing attempt_201105161222_0003_m_000003_0: java.io.IOException: Exception
reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135) at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129) ... 5
more

11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stdout
11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stderr
11/05/16 13:39:33 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_1, Status
: FAILED Error initializing attempt_201105161222_0003_m_000003_1: java.io.IOException: Exception
reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135) at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129) ... 5
more

11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stdout
11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stderr
11/05/16 13:39:36 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_2, Status
: FAILED Error initializing attempt_201105161222_0003_m_000003_2: java.io.IOException: Exception
reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135) at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:125)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129) ... 5
more

11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stdout
11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stderr
11/05/16 13:39:39 INFO mapred.JobClient: Job complete: job_201105161222_0003 11/05/16 13:39:39
INFO mapred.JobClient: Counters: 4 11/05/16 13:39:39 INFO mapred.JobClient: Job Counters 11/05/16
13:39:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=0 11/05/16 13:39:39 INFO mapred.JobClient:
Total time spent by all reduces waiting after reserving slots (ms)=0 11/05/16 13:39:39 INFO
mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 11/05/16
13:39:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0

This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information.  If you have received it in error, please notify the sender
immediately and delete the original.  Any other use of the email by you is prohibited.

Mime
View raw message