hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stuart White" <stuart.whi...@gmail.com>
Subject -libjars with multiple jars broken when client and cluster reside on different OSs?
Date Thu, 11 Dec 2008 17:05:00 GMT
I've written a simple map/reduce job that demonstrates a problem I'm
having.  Please see attached example.

Environment:
  hadoop 0.19.0
  cluster resides across linux nodes
  client resides on cygwin

To recreate the problem I'm seeing, do the following:

- Setup a hadoop cluster on linux

- Perform the remaining steps on cygwin, with a hadoop installation
configured to point to the linux cluster.  (set fs.default.name and
mapred.job.tracker)

- Extract the tarball.  Change into created directory.
  tar xvfz Example.tar.gz
  cd Example

- Edit build.properties, set your hadoop.home appropriately, then
build the example.
  ant

- Load the file Example.in into your dfs
  hadoop dfs -copyFromLocal Example.in Example.in

- Execute the provided shell script, passing it testID 1.
  ./Example.sh 1
  This test does not use -libjars, and it completes successfully.

- Next, execute testID 2.
  ./Example.sh 2
  This test uses -libjars with 1 jarfile (Foo.jar), and it completes
successfully.

- Next, execute testID 3.
  ./Example.sh 3
  This test uses -libjars with 1 jarfile (Bar.jar), and it completes
successfully.

- Next, execute testID 4.
  ./Example.sh 4
  This test uses -libjars with 2 jarfiles (Foo.jar and Bar.jar), and
it fails with a ClassNotFoundException.

This behavior only occurs when calling from cygwin to linux or vice
versa.   If both the cluster and the client reside on either linux or
cygwin, the problem does not occur.

I'm continuing to dig to see what I can figure out, but since I'm very
new to hadoop (started using it this week), I thought I'd go ahead and
throw this out there to see if anyone can help.

Thanks!

Mime
View raw message