hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcos Ortiz Valmaseda <mlor...@uci.cu>
Subject Re: question about using java in streaming mode
Date Sun, 05 Jun 2011 17:59:10 GMT
Why are using Java in streming mode instead use the native Mapper/Reducer code?
Can you show to us the JobTracker's logs?

Regards
----- Mensaje original -----
De: "Siddhartha Jonnalagadda" <sid.kgp@gmail.com>
Para: mapreduce-user@hadoop.apache.org
Enviados: Domingo, 5 de Junio 2011 7:16:08 GMT +01:00 Amsterdam / Berlín / Berna / Roma /
Estocolmo / Viena
Asunto: question about using java in streaming mode

Hi, 


I was able use streaming in hadoop using python for the wordcount program, but created a Mapper
and Reducer in Java since all my code is currently in Java. 
I first tried this: 
echo “foo foo quux labs foo bar quux” |java -cp ~/dummy.jar WCMapper | sort | java -cp
~/dummy.jar WCReducer 

It gave the correct output: 
labs 1 
foo 3 
bar 1 
quux 2 

Then, I installed a single-node cluster in hadoop and tried this: hadoop jar contrib/streaming/hadoop-streaming-0.20.203.0.jar
-mapper “java -cp ~/dummy.jar WCMapper” -reducer “java -cp ~/dummy.jar WCReducer”
-input gutenberg/* -output gutenberg-output -file dummy.jar (by tailoring the python command)


This is the error: 
hadoop@siddhartha-laptop:/usr/local/hadoop$ hadoop jar contrib/streaming/hadoop-streaming-0.20.203.0.jar
-mapper “java -cp ~/dummy.jar WCMapper” -reducer “java -cp ~/dummy.jar WCReducer”
-input gutenberg/* -output gutenberg-output -file dummy.jar 
packageJobJar: [dummy.jar, /app/hadoop/tmp/hadoop-unjar5573454211442575176/] [] /tmp/streamjob6721719460213928092.jar
tmpDir=null 
11/06/04 20:47:15 INFO mapred.FileInputFormat: Total input paths to process : 3 
11/06/04 20:47:15 INFO streaming.StreamJob: getLocalDirs(): [/app/hadoop/tmp/mapred/local]

11/06/04 20:47:15 INFO streaming.StreamJob: Running job: job_201106031901_0039 
11/06/04 20:47:15 INFO streaming.StreamJob: To kill this job, run: 
11/06/04 20:47:15 INFO streaming.StreamJob: /usr/local/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:54311
-kill job_201106031901_0039 
11/06/04 20:47:15 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201106031901_0039

11/06/04 20:47:16 INFO streaming.StreamJob: map 0% reduce 0% 
11/06/04 20:48:00 INFO streaming.StreamJob: map 100% reduce 100% 
11/06/04 20:48:00 INFO streaming.StreamJob: To kill this job, run: 
11/06/04 20:48:00 INFO streaming.StreamJob: /usr/local/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:54311
-kill job_201106031901_0039 
11/06/04 20:48:00 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201106031901_0039

11/06/04 20:48:00 ERROR streaming.StreamJob: Job not successful. Error: NA 
11/06/04 20:48:00 INFO streaming.StreamJob: killJob… 
Streaming Job Failed! 

Any advice? 
Sincerely, 
Siddhartha Jonnalagadda, 
Text mining Researcher, Lnx Research, LLC, Orange, CA 
sjonnalagadda.wordpress.com 







Confidentiality Notice: 

This e-mail message, including any attachments, is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please contact the sender
by reply e-mail and destroy all copies of the original message. 

-- 
Marcos Luís Ortíz Valmaseda
 Software Engineer (Large-Scaled Distributed Systems)
http://marcosluis2186.posterous.com


Mime
View raw message