hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Tice" <mjt...@gmail.com>
Subject Error reading task output for benchmark test TESTDFSIO
Date Thu, 12 May 2011 15:53:03 GMT
Hello,

 

I have a four node hadoop cluster running hadoop v.0.20.2 on CentOS 5.6.
Here is my layout:

 

Name01.hadoop.stage (namenode)

Name02.hadoop.stage (sec namenode / jobtracker)

Data01.hadoop.stage (data node)

Data02.hadoop.stage (data node)

 

When trying to run a benchmark test for this newly-stood up cluster I'm
getting errors.  This is the command (run as the hadoop user on my
name01.hadoop.stage node):

 

# /opt/hadoop/bin/hadoop jar /opt/hadoop/hadoop-0.20.2-test.jar TestDFSIO
-write -nrFiles 1 -fileSize 10

 

Here is the output:

{{BEGIN}}

TestFDSIO.0.0.4

11/05/12 09:35:52 INFO mapred.FileInputFormat: nrFiles = 1

11/05/12 09:35:52 INFO mapred.FileInputFormat: fileSize (MB) = 10

11/05/12 09:35:52 INFO mapred.FileInputFormat: bufferSize = 1000000

11/05/12 09:35:52 INFO mapred.FileInputFormat: creating control file: 10
mega bytes, 1 files

11/05/12 09:35:52 INFO mapred.FileInputFormat: created control files for: 1
files

11/05/12 09:35:52 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.

11/05/12 09:35:52 INFO mapred.FileInputFormat: Total input paths to process
: 1

11/05/12 09:35:52 INFO mapred.JobClient: Running job: job_201105120935_0001

11/05/12 09:35:53 INFO mapred.JobClient:  map 0% reduce 0%

11/05/12 09:35:59 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_0, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:35:59 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_0&filter=stdout

11/05/12 09:35:59 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_0&filter=stderr

11/05/12 09:36:05 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_r_000002_0, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:05 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000002_0&filter=stdout

11/05/12 09:36:05 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000002_0&filter=stderr

11/05/12 09:36:14 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_1, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:14 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_1&filter=stdout

11/05/12 09:36:14 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_1&filter=stderr

11/05/12 09:36:20 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_2, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:20 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_2&filter=stdout

11/05/12 09:36:20 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_2&filter=stderr

11/05/12 09:36:33 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_0, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:33 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_0&filter=stdout

11/05/12 09:36:33 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_0&filter=stderr

11/05/12 09:36:39 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_r_000001_0, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:39 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000001_0&filter=stdout

11/05/12 09:36:39 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000001_0&filter=stderr

11/05/12 09:36:48 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_1, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:48 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_1&filter=stdout

11/05/12 09:36:48 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_1&filter=stderr

11/05/12 09:36:54 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_2, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:54 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_2&filter=stdout

11/05/12 09:36:54 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_2&filter=stderr

11/05/12 09:37:00 INFO mapred.JobClient: Job complete: job_201105120935_0001

11/05/12 09:37:00 INFO mapred.JobClient: Counters: 0

java.io.IOException: Job failed!

                at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)

                at
org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:236)

                at
org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:218)

                at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:354)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver
.java:68)

                at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)

                at
org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:81)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

 

{{END}}

 

Looking at some older posts I thought it may be related to DNS so I added
the hosts/ips of my four-node cluster to /etc/hosts but that didn't help.
Turning DEBUG on on my data01 node I see exceptions similar to:

 

{{BEGIN}}

TestFDSIO.0.0.4

11/05/12 09:35:52 INFO mapred.FileInputFormat: nrFiles = 1

11/05/12 09:35:52 INFO mapred.FileInputFormat: fileSize (MB) = 10

11/05/12 09:35:52 INFO mapred.FileInputFormat: bufferSize = 1000000

11/05/12 09:35:52 INFO mapred.FileInputFormat: creating control file: 10
mega bytes, 1 files

11/05/12 09:35:52 INFO mapred.FileInputFormat: created control files for: 1
files

11/05/12 09:35:52 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.

11/05/12 09:35:52 INFO mapred.FileInputFormat: Total input paths to process
: 1

11/05/12 09:35:52 INFO mapred.JobClient: Running job: job_201105120935_0001

11/05/12 09:35:53 INFO mapred.JobClient:  map 0% reduce 0%

11/05/12 09:35:59 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_0, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:35:59 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_0&filter=stdout

11/05/12 09:35:59 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_0&filter=stderr

11/05/12 09:36:05 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_r_000002_0, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:05 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000002_0&filter=stdout

11/05/12 09:36:05 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000002_0&filter=stderr

11/05/12 09:36:14 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_1, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:14 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_1&filter=stdout

11/05/12 09:36:14 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_1&filter=stderr

11/05/12 09:36:20 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000002_2, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:20 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_2&filter=stdout

11/05/12 09:36:20 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000002_2&filter=stderr

11/05/12 09:36:33 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_0, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:33 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_0&filter=stdout

11/05/12 09:36:33 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_0&filter=stderr

11/05/12 09:36:39 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_r_000001_0, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:39 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000001_0&filter=stdout

11/05/12 09:36:39 WARN mapred.JobClient: Error reading task
outputhttp://data01.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_r_000001_0&filter=stderr

11/05/12 09:36:48 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_1, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:48 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_1&filter=stdout

11/05/12 09:36:48 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_1&filter=stderr

11/05/12 09:36:54 INFO mapred.JobClient: Task Id :
attempt_201105120935_0001_m_000001_2, Status : FAILED

java.io.IOException: Task process exit with nonzero status of 1.

                at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

 

11/05/12 09:36:54 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_2&filter=stdout

11/05/12 09:36:54 WARN mapred.JobClient: Error reading task
outputhttp://data02.hadoop.stage:50060/tasklog?plaintext=true&taskid=attempt
_201105120935_0001_m_000001_2&filter=stderr

11/05/12 09:37:00 INFO mapred.JobClient: Job complete: job_201105120935_0001

11/05/12 09:37:00 INFO mapred.JobClient: Counters: 0

java.io.IOException: Job failed!

                at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)

                at
org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:236)

                at
org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:218)

                at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:354)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver
.java:68)

                at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)

                at
org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:81)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

 

{{END}

 

Another thread mentioned a full FS being a possibility - but my FS usage is
near 0%.  Any help/pointers would be much appreciated.

 

Thanks,

Matt


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message