hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From springring <springr...@126.com>
Subject WholeFileInputFormat with streaming
Date Sat, 02 Mar 2013 08:18:40 GMT
Hi,

    I want to use:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat

    so, I download code from :  
https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
WholeFileInputFormat.java
WholeFileRecordReader.java

    and package the java file with :
package org.apache.hadoop.streaming;

solution A:
copy WholeFileInputFormat.java , WholeFileRecordReader.java  to  hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
then
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
-d WFInputFormatClassNew hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java

there is a lot of error

solution B:
compile the java file   WholeFileInputFormat.java , WholeFileRecordReader.java:
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
-d WFInputFormatClass <two java file>
copy /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar   to <my dir>
then :
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
there is no error, but when I run:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat
.......
there is error:
-inputformat : class not found : org.apache.hadoop.streaming.WholeFileInputFormat

what's wrong with the two solution? or is there any new solution?

thx.

Ring

Mime
View raw message