bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Fryer <dfryer1...@gmail.com>
Subject Mapreduce Jobs Failing
Date Tue, 29 Jul 2014 19:14:29 GMT
Hi Bigtop,

I have a hadoop/mapreduce running on a 4 node bare metal cluster. Each time
when I try to run a job, I get output similar to this:

sudo su hdfs -c "hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 1000"
Number of Maps  = 10
Samples per Map = 1000
14/07/29 13:58:38 WARN mapred.JobConf: The variable mapred.child.ulimit is
no longer used.
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
14/07/29 13:58:40 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/07/29 13:58:40 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/07/29 13:58:41 INFO input.FileInputFormat: Total input paths to process
: 10
14/07/29 13:58:41 INFO mapreduce.JobSubmitter: number of splits:10
14/07/29 13:58:41 WARN conf.Configuration: mapred.jar is deprecated.
Instead, use mapreduce.job.jar
14/07/29 13:58:41 WARN conf.Configuration:
mapred.map.tasks.speculative.execution is deprecated. Instead, use
mapreduce.map.speculative
14/07/29 13:58:41 WARN conf.Configuration: mapred.reduce.tasks is
deprecated. Instead, use mapreduce.job.reduces
14/07/29 13:58:41 WARN conf.Configuration: mapred.output.value.class is
deprecated. Instead, use mapreduce.job.output.value.class
14/07/29 13:58:41 WARN conf.Configuration:
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
14/07/29 13:58:41 WARN conf.Configuration: mapreduce.map.class is
deprecated. Instead, use mapreduce.job.map.class
14/07/29 13:58:41 WARN conf.Configuration: mapred.job.name is deprecated.
Instead, use mapreduce.job.name
14/07/29 13:58:41 WARN conf.Configuration: mapreduce.reduce.class is
deprecated. Instead, use mapreduce.job.reduce.class
14/07/29 13:58:41 WARN conf.Configuration: mapreduce.inputformat.class is
deprecated. Instead, use mapreduce.job.inputformat.class
14/07/29 13:58:41 WARN conf.Configuration: mapred.input.dir is deprecated.
Instead, use mapreduce.input.fileinputformat.inputdir
14/07/29 13:58:41 WARN conf.Configuration: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
14/07/29 13:58:41 WARN conf.Configuration: mapreduce.outputformat.class is
deprecated. Instead, use mapreduce.job.outputformat.class
14/07/29 13:58:41 WARN conf.Configuration: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps
14/07/29 13:58:41 WARN conf.Configuration: mapred.output.key.class is
deprecated. Instead, use mapreduce.job.output.key.class
14/07/29 13:58:41 WARN conf.Configuration: mapred.working.dir is
deprecated. Instead, use mapreduce.job.working.dir
14/07/29 13:58:41 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1406655936930_0009
14/07/29 13:58:42 WARN mapred.JobConf: The variable mapred.child.ulimit is
no longer used.
14/07/29 13:58:42 INFO client.YarnClientImpl: Submitted application
application_1406655936930_0009 to ResourceManager at odin/
192.168.162.164:8032
14/07/29 13:58:42 INFO mapreduce.Job: The url to track the job:
http://odin:20888/proxy/application_1406655936930_0009/
14/07/29 13:58:42 INFO mapreduce.Job: Running job: job_1406655936930_0009
14/07/29 13:58:46 INFO mapreduce.Job: Job job_1406655936930_0009 running in
uber mode : false
14/07/29 13:58:46 INFO mapreduce.Job:  map 0% reduce 0%
14/07/29 13:58:59 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000001_0, Status : FAILED
14/07/29 13:58:59 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000000_0, Status : FAILED
14/07/29 13:59:01 INFO mapreduce.Job:  map 30% reduce 0%
14/07/29 13:59:01 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000006_0, Status : FAILED
14/07/29 13:59:01 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000002_0, Status : FAILED
14/07/29 13:59:01 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000008_0, Status : FAILED
14/07/29 13:59:01 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000009_0, Status : FAILED
14/07/29 13:59:02 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000003_0, Status : FAILED
14/07/29 13:59:02 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000005_0, Status : FAILED
14/07/29 13:59:02 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000007_0, Status : FAILED
14/07/29 13:59:02 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000004_0, Status : FAILED
14/07/29 13:59:03 INFO mapreduce.Job:  map 0% reduce 0%
14/07/29 13:59:06 INFO mapreduce.Job:  map 10% reduce 0%
14/07/29 13:59:07 INFO mapreduce.Job:  map 20% reduce 0%
14/07/29 13:59:12 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000000_1, Status : FAILED
14/07/29 13:59:12 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000001_1, Status : FAILED
14/07/29 13:59:14 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000002_1, Status : FAILED
14/07/29 13:59:14 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000006_1, Status : FAILED
14/07/29 13:59:15 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000009_1, Status : FAILED
14/07/29 13:59:15 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000008_1, Status : FAILED
14/07/29 13:59:16 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000005_1, Status : FAILED
14/07/29 13:59:16 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000007_1, Status : FAILED
14/07/29 13:59:19 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_r_000000_0, Status : FAILED
14/07/29 13:59:20 INFO mapreduce.Job:  map 30% reduce 0%
14/07/29 13:59:26 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000001_2, Status : FAILED
14/07/29 13:59:26 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000000_2, Status : FAILED
14/07/29 13:59:28 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000002_2, Status : FAILED
14/07/29 13:59:28 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000006_2, Status : FAILED
14/07/29 13:59:28 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000007_2, Status : FAILED
14/07/29 13:59:29 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000008_2, Status : FAILED
14/07/29 13:59:29 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_m_000009_2, Status : FAILED
14/07/29 13:59:33 INFO mapreduce.Job:  map 40% reduce 0%
14/07/29 13:59:34 INFO mapreduce.Job: Task Id :
attempt_1406655936930_0009_r_000000_1, Status : FAILED
14/07/29 13:59:40 INFO mapreduce.Job:  map 60% reduce 0%
14/07/29 13:59:40 INFO mapreduce.Job: Job job_1406655936930_0009 failed
with state FAILED due to: Task failed task_1406655936930_0009_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/07/29 13:59:40 WARN mapred.JobConf: The variable mapred.child.ulimit is
no longer used.
14/07/29 13:59:41 INFO mapreduce.Job: Counters: 34
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=300264
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1040
HDFS: Number of bytes written=0
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Failed map tasks=27
Failed reduce tasks=2
Launched map tasks=35
Launched reduce tasks=3
Other local map tasks=25
Data-local map tasks=2
Rack-local map tasks=8
Total time spent by all maps in occupied slots (ms)=351421
Total time spent by all reduces in occupied slots (ms)=24148
Map-Reduce Framework
Map input records=4
Map output records=8
Map output bytes=72
Map output materialized bytes=112
Input split bytes=568
Combine input records=0
Spilled Records=8
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=60
CPU time spent (ms)=1270
Physical memory (bytes) snapshot=1694736384
Virtual memory (bytes) snapshot=7109738496
Total committed heap usage (bytes)=1664090112
File Input Format Counters
Bytes Read=472
Job Finished in 60.202 seconds
java.io.FileNotFoundException: File does not exist:
hdfs://odin:17020/user/hdfs/QuasiMonteCarlo_1406660318881_1358196499/out/reduce-out
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:815)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Does anyone have any idea what would cause this?

-David Fryer

Mime
View raw message