hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Simpson" <ssimp...@InterchangeUSA.com>
Subject What I had to do to get the Jython WordCount.py example working
Date Wed, 30 Aug 2006 21:05:44 GMT
Here are the steps I had to do to get the WordCount.py file working
under Jython. This will save you some grief if you need to do the same
thing.
 
1. Install Jython 2.1 with "java -cp . jython_21" assuming jython.class
is in the current directory.
2. I am running JDK 1.5 so I had to use Jython 2.2alpha1 which only
comes in source form. This means that you need to compile it.
Unfortunately, the build.xml didn't work for me and I made the following
diffs:
 
*** build.xml~  2006-08-28 12:12:08.000000000 -0700
--- build.xml   2006-08-28 16:28:00.000000000 -0700
***************
*** 135,142 ****
          <property file="${basedir}/ant.properties" />
 
          <property name="build.compiler" value="modern" />
!         <property name="jdk.target.version" value="1.2" />
!         <property name="jdk.source.version" value="1.3" />
          <property name="deprecation" value="off" />
          <property name="debug" value="off" />
          <property name="optimize" value="on" />
--- 135,142 ----
          <property file="${basedir}/ant.properties" />
 
          <property name="build.compiler" value="modern" />
!         <property name="jdk.target.version" value="1.5" />
!         <property name="jdk.source.version" value="1.5" />
          <property name="deprecation" value="off" />
          <property name="debug" value="off" />
          <property name="optimize" value="on" />
***************
*** 346,352 ****
 
      <!-- compile java sources -->
      <target name="compile" depends="checkout, parser">
!         <javac srcdir="${source.dir}/"
                 destdir="${output.dir}/"
                 target="${jdk.target.version}"
                 source="${jdk.source.version}"
--- 347,354 ----
 
      <!-- compile java sources -->
      <target name="compile" depends="checkout, parser">
!         <javac srcdir="${source.dir}/src/java"
                 destdir="${output.dir}/"
                 target="${jdk.target.version}"
                 source="${jdk.source.version}"
***************
*** 406,418 ****
                 nowarn="${nowarn}"
          />
 
!         <copy file="org/python/modules/ucnhash.dat"
                todir="${output.dir}/org/python/modules"
                preservelastmodified="true"
          />
 
          <copy todir="${output.dir}/com" preservelastmodified="true">
!             <fileset dir="${source.dir}/com">
                  <include name="**/*.properties"
if="java2collection.present" />
              </fileset>
          </copy>
--- 408,420 ----
                 nowarn="${nowarn}"
          />
 
!         <copy
file="${source.dir}/src/java/org/python/modules/ucnhash.dat"
                todir="${output.dir}/org/python/modules"
                preservelastmodified="true"
          />
 
          <copy todir="${output.dir}/com" preservelastmodified="true">
!             <fileset dir="${source.dir}/src/java/com">
                  <include name="**/*.properties"
if="java2collection.present" />
              </fileset>
          </copy>

My "ant.properties" looks like
 
javaccHome=/apps/build/javacc-4.0
python.home=/apps/build/Python-2.4.3
ht2html.dir=/apps/build/ht2html-2.0
readline.jar=/apps/build/readline-5.1

but you'll need to edit your own. Do an "ant" and you should get the
distribution in the "dist" directory.
3. Delete the "jython.jar" and "Lib" and "Tools" directories from
whereever you installed Jython 2.1. Copy the contents of "dist" into
that directory (i.e., replace the "jython.jar", "Lib", and "Tools" with
the 2.2alpha1 binaries).
4. The existing "jython" script in the jython installation directory has
a bug where an empty CLASSPATH element causes an error. I've replaced
that script with
 
#!/bin/sh
########################################################################
#######
#
# This file generated by Jython installer
# Created on XXX by root
 
if [ -z "$CLASSPATH" ]; then
    "/apps/java1.5/jre/bin/java"
-Dpython.home="/apps/local/jython-2.2alpha1" -classpath
"/apps/local/jython-2.2alpha1/jython.jar" "org.python.util.jython" "$@"
else
    "/apps/java1.5/jre/bin/java"
-Dpython.home="/apps/local/jython-2.2alpha1" -classpath
"/apps/local/jython-2.2alpha1/jython.jar:$CLASSPATH"
"org.python.util.jython" "$@"
fi

You'll need to edit the paths to taste.
5. Compile WordCount.py using the supplied "compile" script.  This will
generate a "wc.jar".
6. Run the command in Hadoop with "hadoop jar wc.jar dir1 dir2" where
dir1 is a directory containing files and dir2 doesn't exist. I'm running
mine out of dfs but you could use a local file system too depending on
how you have Hadoop configured. You can put a directory in dfs with
"hadoop dfs -put dir1 dir1". Again, the dir2 directory shouldn't exist.
 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message