hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From amcn...@mcnabbs.org (Andrew McNabb)
Subject HadoopStreaming
Date Tue, 17 Oct 2006 20:12:06 GMT
HadoopStreaming looks really cool, and I'm trying it for the first time.
I'm obviously doing something wrong, but I have no clue what.

I made a goofy little wordcount mapper and reducer in Python, and I'm
running HadoopStreaming with the following alias:

alias hadoop-streaming='/home/amcnabb/hadoop/bin/hadoop jar /home/amcnabb/hadoop/build/hadoop-streaming.jar'

Here is the job I ran, after doing "hadoop dfs -put kjv kjv".  From the output,
it is clear that: "Job not Successful!" However, I have no idea of
what's causing the problem.  Am I messing up something obvious?  Where
should I look to see what's really happening?  Thanks.

amcnabb@prodigy:~/svn/mrpso/python% hadoop-streaming -mapper "/usr/bin/python /home/amcnabb/svn/mrpso/python/mapper.py"
-reducer "/usr/bin/python /home/amcnabb/svn/mrpso/python/reducer.py" -input kjv -output kjvout
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/mapred-default.xml
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml
06/10/17 14:09:11 INFO ipc.Client: org.apache.hadoop.io.ObjectWritable ConnectionCuller maxidletime=1000ms:
starting
packageJobJar: [/tmp/hadoop-unjar29550] [] /tmp/streamjob29551.jar
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml
06/10/17 14:09:11 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-amcnabb/mapred/local]
06/10/17 14:09:11 INFO streaming.StreamJob: Running job: job_0013
06/10/17 14:09:11 INFO streaming.StreamJob: To kill this job, run:
06/10/17 14:09:11 INFO streaming.StreamJob: /home/amcnabb/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=prodigy:50006
-kill job_0013
06/10/17 14:09:11 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013
06/10/17 14:09:12 INFO streaming.StreamJob:  map 100%  reduce 100%
06/10/17 14:09:12 INFO streaming.StreamJob: To kill this job, run:
06/10/17 14:09:12 INFO streaming.StreamJob: /home/amcnabb/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=prodigy:50006
-kill job_0013
06/10/17 14:09:12 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013
06/10/17 14:09:12 INFO streaming.StreamJob: killJob...
Exception in thread "main" java.io.IOException: Job not Successful!
	at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:558)
	at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:63)
	at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:29)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:137)
amcnabb@prodigy:~/svn/mrpso/python%

-- 
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868

Mime
View raw message