hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hassaan Khan <hass...@cooliris.com>
Subject Multinode cluster setup issues
Date Wed, 28 Oct 2009 15:41:32 GMT
I'm running Hadoop 0.20.1+133 (Cloudera distro)
I tried setting up a multi-node Hadoop cluster and on executing the command:
hadoop jar /usr/lib/hadoop/hadoop-0.20.1+133-examples.jar grep input output
'dfs[a-z.]+'
I get:

09/10/27 20:39:21 INFO mapred.FileInputFormat: Total input paths to process
: 5
09/10/27 20:39:21 INFO mapred.JobClient: Running job: job_200910272023_0002
09/10/27 20:39:22 INFO mapred.JobClient:  map 0% reduce 0%
09/10/27 20:39:30 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000006_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:39:30 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_0&filter=stdout
09/10/27 20:39:30 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_0&filter=stderr
09/10/27 20:39:36 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000020_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:39:36 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_0&filter=stdout
09/10/27 20:39:36 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_0&filter=stderr
09/10/27 20:39:42 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000006_1, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:39:42 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_1&filter=stdout
09/10/27 20:39:42 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_1&filter=stderr
09/10/27 20:39:48 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000020_1, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:39:48 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_1&filter=stdout
09/10/27 20:39:48 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_1&filter=stderr
09/10/27 20:39:57 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000006_2, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:39:57 WARN mapred.JobClient: Error reading task outputhttp://
anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_2&filter=stdout
09/10/27 20:39:57 WARN mapred.JobClient: Error reading task outputhttp://
anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_2&filter=stderr
09/10/27 20:40:03 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000020_2, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:40:03 WARN mapred.JobClient: Error reading task outputhttp://
anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_2&filter=stdout
09/10/27 20:40:03 WARN mapred.JobClient: Error reading task outputhttp://
anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_2&filter=stderr
09/10/27 20:40:15 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000005_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:40:15 WARN mapred.JobClient: Error reading task outputhttp://
anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_0&filter=stdout
09/10/27 20:40:15 WARN mapred.JobClient: Error reading task outputhttp://
anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_0&filter=stderr
09/10/27 20:40:21 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000019_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:40:21 WARN mapred.JobClient: Error reading task outputhttp://
anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_0&filter=stdout
09/10/27 20:40:21 WARN mapred.JobClient: Error reading task outputhttp://
anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_0&filter=stderr
09/10/27 20:40:30 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000005_1, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:40:30 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_1&filter=stdout
09/10/27 20:40:30 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_1&filter=stderr
09/10/27 20:40:36 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000019_1, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:40:36 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_1&filter=stdout
09/10/27 20:40:36 WARN mapred.JobClient: Error reading task outputhttp://
anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_1&filter=stderr
09/10/27 20:40:42 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_m_000005_2, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:40:42 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_2&filter=stdout
09/10/27 20:40:42 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_2&filter=stderr
09/10/27 20:40:48 INFO mapred.JobClient: Task Id :
attempt_200910272023_0002_r_000019_2, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458)

09/10/27 20:40:48 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_2&filter=stdout
09/10/27 20:40:48 WARN mapred.JobClient: Error reading task outputhttp://
anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_2&filter=stderr
09/10/27 20:40:57 INFO mapred.JobClient: Job complete: job_200910272023_0002
09/10/27 20:40:57 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1293)
    at org.apache.hadoop.examples.Grep.run(Grep.java:69)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.examples.Grep.main(Grep.java:93)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:185)



Based upon a post I read to a similar issue, I changed my /etc/hosts file
to:

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1        localhost.localdomain localhost
::1        localhost6.localdomain6 localhost6
10.50.65.61        anza1.eng.blah.com anza1
10.50.65.62        anza2.eng.blah.com anza2
10.50.65.63        anza3.eng.blah.com anza3
10.50.65.64        anza4.eng.blah.com anza4
10.50.65.65        anza5.eng.blah.com anza5



Also, when I look at:
/var/log/hadoop/userlogs/attempt_200910271659_0007_r_000019_0 on a slave
STDOUT:
Error occurred during initialization of VM
Could not reserve enough space for object heap
STDERR:
Could not create the Java virtual machine.


My slaves are running on boxes with 8GB or RAM and under:
JAVA_HEAP_MAX=-Xmx1000m

And under mapred-site.xml:
<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx2048m</value>
  </property>


I can't figure out why the slaves are failing?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message