hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From oj...@doc.ic.ac.uk
Subject Running hadoop job from Gui
Date Fri, 27 Jul 2007 10:35:34 GMT
Hi,

I've got a GUI based program that I'm working on, and I'm trying to  
add some funcionality to it where it runs a map reduce job on hadoop.  
For the moment I am assuming that anyone who is running the program  
will be running it on a machine with a hadoop system running, although  
later I would like for them to be able to point it at a different  
machine on the network.

So, everything works fine when I run the GUI from eclipse (I also have  
a hadoop project in there BTW). However, when I run the GUI from the  
command line I get the following exception:

java.io.IOException: No FileSystem for scheme: file
         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
         at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)
         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:91)
         at pipe.dataLayer.calculations.RTAMapRed.launch(RTAMapRed.java:293)
and so on......

The bit of code that causes it is (the offending line is marked ****):
		conf = new Configuration();
	    jobConf = new JobConf(conf, RTAMapRed.class);

	    jobConf.setJobName("rta");

//	  turn off speculative execution, because DFS doesn't handle
	    // multiple writers to the same file.
	    jobConf.setSpeculativeExecution(false);

	    jobConf.setInputFormat(EulerSequenceFileInputFormat.class);

	    jobConf.setOutputKeyClass(ComplexWritable.class);
	    jobConf.setOutputValueClass(DoubleWritable.class);
	    jobConf.setOutputFormat(SequenceFileOutputFormat.class);

	    jobConf.setMapperClass(RTAMapper.class);
	    jobConf.setReducerClass(RTAReducer.class);

	    tmpDir = new Path("rtamapred");
	    inDir = new Path(tmpDir, "in");
	    outDir = new Path(tmpDir, "out");
	    FileSystem fileSys = FileSystem.get(jobConf); //****This line here!!
	    fileSys.delete(tmpDir);
	    if (!fileSys.mkdirs(inDir)) {
	      throw new IOException("Mkdirs failed to create " + inDir.toString());
	    }

I'm running the GUI with the following script to include my hadoop  
installation (its a developer one) on the classpath. I was wondering  
if I've missed anything obvious off?

CLASSPATH="${HADOOP_CONF_DIR}"
CLASSPATH=${CLASSPATH}:bin
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
   CLASSPATH=${CLASSPATH}:/home/ollie/workspace/hadoop/build/classes
   CLASSPATH=${CLASSPATH}:/home/ollie/workspace/hadoop/build
CLASSPATH=${CLASSPATH}:$HADOOP_INSTALL/build/test/classes
for f in /home/ollie/workspace/hadoop/lib/*.jar; do
   CLASSPATH=${CLASSPATH}:$f;
done

for f in $HADOOP_INSTALL/lib/jetty-ext/*.jar; do
   CLASSPATH=${CLASSPATH}:$f;
done
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.dir=$HADOOP_LOG_DIR"
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.file=$HADOOP_LOGFILE"
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.home.dir=$HADOOP_HOME"
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.id.str=$HADOOP_IDENT_STRING"
HADOOP_OPTS="$HADOOP_OPTS  
-Dhadoop.root.logger=${HADOOP_ROOT_LOGGER:-INFO,console}"
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
   HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
fi
exec java $HADOOP_OPTS -classpath "$CLASSPATH" RunGui "$@"
(I've tried it with and without the $HADOOP_OPTS set )

Thanks very much in advance for any help offered and apologies for the  
information overload!

Cheers,
Ollie


Mime
View raw message