hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Margusja <mar...@roo.ee>
Subject Task failure in slave node
Date Thu, 11 Jul 2013 12:44:44 GMT
Hi

I have tow nodes:
n1 (master, salve) and n2 (slave)

after set up I ran wordcount example and it worked fine:
[hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-examples-1.0.4.jar 
wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to 
process : 7
13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop 
library
13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded
13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015
13/07/11 15:30:45 INFO mapred.JobClient:  map 0% reduce 0%
13/07/11 15:31:03 INFO mapred.JobClient:  map 42% reduce 0%
13/07/11 15:31:06 INFO mapred.JobClient:  map 57% reduce 0%
13/07/11 15:31:09 INFO mapred.JobClient:  map 71% reduce 0%
13/07/11 15:31:15 INFO mapred.JobClient:  map 100% reduce 0%
13/07/11 15:31:18 INFO mapred.JobClient:  map 100% reduce 23%
13/07/11 15:31:27 INFO mapred.JobClient:  map 100% reduce 100%
13/07/11 15:31:32 INFO mapred.JobClient: Job complete: job_201307111355_0015
13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30
13/07/11 15:31:32 INFO mapred.JobClient:   Job Counters
13/07/11 15:31:32 INFO mapred.JobClient:     Launched reduce tasks=1
13/07/11 15:31:32 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=67576
13/07/11 15:31:32 INFO mapred.JobClient:     Total time spent by all 
reduces waiting after reserving slots (ms)=0
13/07/11 15:31:32 INFO mapred.JobClient:     Total time spent by all 
maps waiting after reserving slots (ms)=0
13/07/11 15:31:32 INFO mapred.JobClient:     Rack-local map tasks=3
13/07/11 15:31:32 INFO mapred.JobClient:     Launched map tasks=7
13/07/11 15:31:32 INFO mapred.JobClient:     Data-local map tasks=4
13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992
13/07/11 15:31:32 INFO mapred.JobClient:   File Output Format Counters
13/07/11 15:31:32 INFO mapred.JobClient:     Bytes Written=1412505
13/07/11 15:31:32 INFO mapred.JobClient:   FileSystemCounters
13/07/11 15:31:32 INFO mapred.JobClient:     FILE_BYTES_READ=5414195
13/07/11 15:31:32 INFO mapred.JobClient:     HDFS_BYTES_READ=6950820
13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993
13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505
13/07/11 15:31:32 INFO mapred.JobClient:   File Input Format Counters
13/07/11 15:31:32 INFO mapred.JobClient:     Bytes Read=6950001
13/07/11 15:31:32 INFO mapred.JobClient:   Map-Reduce Framework
13/07/11 15:31:32 INFO mapred.JobClient:     Map output materialized 
bytes=3157469
13/07/11 15:31:32 INFO mapred.JobClient:     Map input records=137146
13/07/11 15:31:32 INFO mapred.JobClient:     Reduce shuffle bytes=2904836
13/07/11 15:31:32 INFO mapred.JobClient:     Spilled Records=594764
13/07/11 15:31:32 INFO mapred.JobClient:     Map output bytes=11435849
13/07/11 15:31:32 INFO mapred.JobClient:     Total committed heap usage 
(bytes)=1128136704
13/07/11 15:31:32 INFO mapred.JobClient:     CPU time spent (ms)=18230
13/07/11 15:31:32 INFO mapred.JobClient:     Combine input records=1174991
13/07/11 15:31:32 INFO mapred.JobClient:     SPLIT_RAW_BYTES=819
13/07/11 15:31:32 INFO mapred.JobClient:     Reduce input records=218990
13/07/11 15:31:32 INFO mapred.JobClient:     Reduce input groups=128513
13/07/11 15:31:32 INFO mapred.JobClient:     Combine output records=218990
13/07/11 15:31:32 INFO mapred.JobClient:     Physical memory (bytes) 
snapshot=1179656192
13/07/11 15:31:32 INFO mapred.JobClient:     Reduce output records=128513
13/07/11 15:31:32 INFO mapred.JobClient:     Virtual memory (bytes) 
snapshot=22992117760
13/07/11 15:31:32 INFO mapred.JobClient:     Map output records=1174991

from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were 
used without any errors.

Problems appear if I try to use following commands in master (n1):

[hduser@n1 ~]$hadoop jar 
mahout-distribution-0.7/mahout-examples-0.7-job.jar 
org.apache.mahout.classifier.df.mapreduce.BuildForest 
-Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv -ds 
testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[file:/usr/local/hadoop-1.0.4/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred implementation
13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest...
13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set.  User 
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to 
process : 1
13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop 
library
13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded
13/07/11 15:36:50 INFO mapred.JobClient: Running job: job_201307111355_0016
13/07/11 15:36:51 INFO mapred.JobClient:  map 0% reduce 0%
13/07/11 15:37:09 INFO mapred.JobClient:  map 7% reduce 0%
13/07/11 15:37:11 INFO mapred.JobClient: Task Id : 
attempt_201307111355_0016_m_000002_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
         at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
         at 
org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:415)
         at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
         at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: 
org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
         at java.security.AccessController.doPrivileged(Native Method)
         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at java.lang.Class.forName0(Native Method)
         at java.lang.Class.forName(Class.java:270)
         at 
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
         at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
         ... 8 more

...
...
...

13/07/11 15:37:37 INFO mapred.JobClient:  map 61% reduce 0%
13/07/11 15:37:46 INFO mapred.JobClient:  map 76% reduce 0%
13/07/11 15:37:55 INFO mapred.JobClient:  map 92% reduce 0%
13/07/11 15:38:01 INFO mapred.JobClient:  map 100% reduce 0%
13/07/11 15:38:06 INFO mapred.JobClient: Job complete: job_201307111355_0016
13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20
13/07/11 15:38:06 INFO mapred.JobClient:   Job Counters
13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724
13/07/11 15:38:06 INFO mapred.JobClient:     Total time spent by all 
reduces waiting after reserving slots (ms)=0
13/07/11 15:38:06 INFO mapred.JobClient:     Total time spent by all 
maps waiting after reserving slots (ms)=0
13/07/11 15:38:06 INFO mapred.JobClient:     Rack-local map tasks=13
13/07/11 15:38:06 INFO mapred.JobClient:     Launched map tasks=22
13/07/11 15:38:06 INFO mapred.JobClient:     Data-local map tasks=9
13/07/11 15:38:06 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
13/07/11 15:38:06 INFO mapred.JobClient:   File Output Format Counters
13/07/11 15:38:06 INFO mapred.JobClient:     Bytes Written=184520
13/07/11 15:38:06 INFO mapred.JobClient:   FileSystemCounters
13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037
13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520
13/07/11 15:38:06 INFO mapred.JobClient:   File Input Format Counters
13/07/11 15:38:06 INFO mapred.JobClient:     Bytes Read=22952913
13/07/11 15:38:06 INFO mapred.JobClient:   Map-Reduce Framework
13/07/11 15:38:06 INFO mapred.JobClient:     Map input records=204091
13/07/11 15:38:06 INFO mapred.JobClient:     Physical memory (bytes) 
snapshot=1402363904
13/07/11 15:38:06 INFO mapred.JobClient:     Spilled Records=0
13/07/11 15:38:06 INFO mapred.JobClient:     CPU time spent (ms)=26640
13/07/11 15:38:06 INFO mapred.JobClient:     Total committed heap usage 
(bytes)=1333985280
13/07/11 15:38:06 INFO mapred.JobClient:     Virtual memory (bytes) 
snapshot=37349126144
13/07/11 15:38:06 INFO mapred.JobClient:     Map output records=1
13/07/11 15:38:06 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1573
13/07/11 15:38:07 INFO common.HadoopUtil: Deleting 
hdfs://master:54310/user/hduser/bal_ee_2009_out
13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977
13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700
13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700
13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17
13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in: 
bal_ee_2009_out/forest.seq

 From web console I can see that n1 tasks are fine but n2 tasks failed:

java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
	at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.classifier.df.mapreduce.partial.Step1Mapper
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
	... 8 more


But If I run same command from n2
[hduser@n2 ~]$ hadoop jar 
mahout-distribution-0.7/mahout-examples-0.7-job.jar 
org.apache.mahout.classifier.df.mapreduce.BuildForest 
-Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv -ds 
testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1

13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred implementation
13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest...
13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to 
process : 1
13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop 
library
13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded
13/07/11 15:41:21 INFO mapred.JobClient: Running job: job_201307111355_0017
13/07/11 15:41:22 INFO mapred.JobClient:  map 0% reduce 0%
13/07/11 15:41:43 INFO mapred.JobClient:  map 7% reduce 0%
13/07/11 15:41:46 INFO mapred.JobClient:  map 15% reduce 0%
13/07/11 15:41:49 INFO mapred.JobClient:  map 38% reduce 0%
13/07/11 15:41:52 INFO mapred.JobClient:  map 46% reduce 0%
13/07/11 15:42:01 INFO mapred.JobClient:  map 61% reduce 0%
13/07/11 15:42:02 INFO mapred.JobClient:  map 76% reduce 0%
13/07/11 15:42:09 INFO mapred.JobClient:  map 84% reduce 0%
13/07/11 15:42:13 INFO mapred.JobClient:  map 100% reduce 0%
13/07/11 15:42:19 INFO mapred.JobClient: Job complete: job_201307111355_0017
13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20
13/07/11 15:42:19 INFO mapred.JobClient:   Job Counters
13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911
13/07/11 15:42:19 INFO mapred.JobClient:     Total time spent by all 
reduces waiting after reserving slots (ms)=0
13/07/11 15:42:19 INFO mapred.JobClient:     Total time spent by all 
maps waiting after reserving slots (ms)=0
13/07/11 15:42:19 INFO mapred.JobClient:     Rack-local map tasks=6
13/07/11 15:42:19 INFO mapred.JobClient:     Launched map tasks=13
13/07/11 15:42:19 INFO mapred.JobClient:     Data-local map tasks=7
13/07/11 15:42:19 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
13/07/11 15:42:19 INFO mapred.JobClient:   File Output Format Counters
13/07/11 15:42:19 INFO mapred.JobClient:     Bytes Written=184956
13/07/11 15:42:19 INFO mapred.JobClient:   FileSystemCounters
13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612
13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026
13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956
13/07/11 15:42:19 INFO mapred.JobClient:   File Input Format Counters
13/07/11 15:42:19 INFO mapred.JobClient:     Bytes Read=22952913
13/07/11 15:42:19 INFO mapred.JobClient:   Map-Reduce Framework
13/07/11 15:42:19 INFO mapred.JobClient:     Map input records=204091
13/07/11 15:42:19 INFO mapred.JobClient:     Physical memory (bytes) 
snapshot=1370255360
13/07/11 15:42:19 INFO mapred.JobClient:     Spilled Records=0
13/07/11 15:42:19 INFO mapred.JobClient:     CPU time spent (ms)=27990
13/07/11 15:42:19 INFO mapred.JobClient:     Total committed heap usage 
(bytes)=1292500992
13/07/11 15:42:19 INFO mapred.JobClient:     Virtual memory (bytes) 
snapshot=37360082944
13/07/11 15:42:19 INFO mapred.JobClient:     Map output records=1
13/07/11 15:42:19 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1573
13/07/11 15:42:20 INFO common.HadoopUtil: Deleting 
hdfs://master:54310/user/hduser/bal_ee_2009_out
13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386
13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698
13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698
13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16
13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in: 
bal_ee_2009_out/forest.seq

Both (n1 and n2) are used and from web console I can see that there are 
no errors.

Is there any explanations why I am getting errors when I run command 
from master?


-- 
Regards, Margus (Margusja) Roo
+372 51 48 780
http://margus.roo.ee
skype: margusja
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
BjM8j36yJvoBVsfOHQIDAQAB
-----END PUBLIC KEY-----


Mime
View raw message