hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yoram Arnon" <yar...@yahoo-inc.com>
Subject RE: HadoopStreaming
Date Tue, 17 Oct 2006 22:52:59 GMT
Try changing your command to read  

hadoop-streaming \
-mapper "/usr/bin/python mapper.py" \
-file "/home/amcnabb/svn/mrpso/python/mapper.py" \
-reducer "/usr/bin/python reducer.py" \
-file "/home/amcnabb/svn/mrpso/python/reducer.py"  \
-input kjv \
-output kjvout

I assume kjv is a file and kjvout is a directory - they should be.
I also assume /usr/bin/python is the path to python *on the cluster
machines*. Otherwise, you can do
-mapper "python mapper.py" -file /usr/bin/python -file
/home/amcnabb/svn/mrpso/python/mapper.py
I recommend adding -jobconf mapred.job.name="kjv", to make the jobtracker
history more readable.

Yoram

> -----Original Message-----
> From: Andrew McNabb [mailto:amcnabb@mcnabbs.org] 
> Sent: Tuesday, October 17, 2006 1:12 PM
> To: hadoop-user@lucene.apache.org
> Subject: HadoopStreaming
> 
> HadoopStreaming looks really cool, and I'm trying it for the 
> first time.
> I'm obviously doing something wrong, but I have no clue what.
> 
> I made a goofy little wordcount mapper and reducer in Python, and I'm
> running HadoopStreaming with the following alias:
> 
> alias hadoop-streaming='/home/amcnabb/hadoop/bin/hadoop jar 
> /home/amcnabb/hadoop/build/hadoop-streaming.jar'
> 
> Here is the job I ran, after doing "hadoop dfs -put kjv kjv". 
>  From the output,
> it is clear that: "Job not Successful!" However, I have no idea of
> what's causing the problem.  Am I messing up something obvious?  Where
> should I look to see what's really happening?  Thanks.
> 
> amcnabb@prodigy:~/svn/mrpso/python% hadoop-streaming -mapper 
> "/usr/bin/python /home/amcnabb/svn/mrpso/python/mapper.py" 
> -reducer "/usr/bin/python 
> /home/amcnabb/svn/mrpso/python/reducer.py" -input kjv -output kjvout
> 06/10/17 14:09:11 INFO conf.Configuration: parsing 
> file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml
> 06/10/17 14:09:11 INFO conf.Configuration: parsing 
> file:/home/amcnabb/hadoop-0.6.2/conf/mapred-default.xml
> 06/10/17 14:09:11 INFO conf.Configuration: parsing 
> file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml
> 06/10/17 14:09:11 INFO ipc.Client: 
> org.apache.hadoop.io.ObjectWritable ConnectionCuller 
> maxidletime=1000ms: starting
> packageJobJar: [/tmp/hadoop-unjar29550] [] /tmp/streamjob29551.jar
> 06/10/17 14:09:11 INFO conf.Configuration: parsing 
> file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml
> 06/10/17 14:09:11 INFO conf.Configuration: parsing 
> file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml
> 06/10/17 14:09:11 INFO streaming.StreamJob: getLocalDirs(): 
> [/tmp/hadoop-amcnabb/mapred/local]
> 06/10/17 14:09:11 INFO streaming.StreamJob: Running job: job_0013
> 06/10/17 14:09:11 INFO streaming.StreamJob: To kill this job, run:
> 06/10/17 14:09:11 INFO streaming.StreamJob: 
> /home/amcnabb/hadoop/bin/../bin/hadoop job  
> -Dmapred.job.tracker=prodigy:50006 -kill job_0013
> 06/10/17 14:09:11 INFO streaming.StreamJob: Tracking URL: 
> http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013
> 06/10/17 14:09:12 INFO streaming.StreamJob:  map 100%  reduce 100%
> 06/10/17 14:09:12 INFO streaming.StreamJob: To kill this job, run:
> 06/10/17 14:09:12 INFO streaming.StreamJob: 
> /home/amcnabb/hadoop/bin/../bin/hadoop job  
> -Dmapred.job.tracker=prodigy:50006 -kill job_0013
> 06/10/17 14:09:12 INFO streaming.StreamJob: Tracking URL: 
> http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013
> 06/10/17 14:09:12 INFO streaming.StreamJob: killJob...
> Exception in thread "main" java.io.IOException: Job not Successful!
> 	at 
> org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(Stre
> amJob.java:558)
> 	at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:63)
> 	at 
> org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreami
> ng.java:29)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess
> orImpl.java:39)
> 	at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:585)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:137)
> amcnabb@prodigy:~/svn/mrpso/python%
> 
> -- 
> Andrew McNabb
> http://www.mcnabbs.org/andrew/
> PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868
> 


Mime
View raw message