hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Hadoop installation on Windows
Date Tue, 16 Nov 2010 12:24:19 GMT
On 13/11/10 23:34, Christopher Worley wrote:
> Thanks for the advice, guys.  I found this tutorial that covers
> installation of 0.21.0:
> http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/
> The author suggests adding the "CLASSPATH=`cygpath -wp "$CLASSPATH"`"
> to bin/hadoop-config.sh just like you suggested.  I made that change
> and then checked the version.  Here's what I got:
> $ bin/hadoop version
> cygwin warning:
>    MS-DOS style path detected: C:\cygwin\usr\local\hadoop-0.21.0\/build/native
>    Preferred POSIX equivalent is: /usr/local/hadoop-0.21.0/build/native
>    CYGWIN environment variable option "nodosfilewarning" turns off this warning.
>    Consult the user's guide for more details about POSIX paths:
>      http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
> Hadoop 0.21.0
> Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -
> r 985326
> Compiled by tomwhite on Tue Aug 17 01:02:28 EDT 2010
>  From source with checksum a1aeb15b4854808d152989ba76f90fac
> It gives a warning on the path format, but it appears to work--it
> gives the correct Hadoop version.
> When I try to do the next step, formatting the namenode, I get the
> following exceptions:
> http://pastebin.com/YhZ8JZjG
> When I try to run "bin/start-dfs.sh" or "bin/start-mapred.sh" I get
> "Hadoop common not found".
> I appreciate any help.

If you can get the classpath printed out, it may be that 
hadoop-common.jar isn't on it.

As an aside, based on my experience on other projects, the shell scripts 
to start java apps are an extreme source of pain, for various reasons
  -shell script engine variations on Unix platforms (bash, unix sh)
  -bash isn't that good a language for complex development (at least by 
java programmers)
  -cygwin causes extra confusion
  -zero/not enough testing of the scripts in the unit tests

For the limited number of lines, they generate a lot of support calls.

I prefer using python as a launcher, as it's more consistent. Hadoop may 
benefit from a python launcher, although Hadoop will probably still 
expect cygwin on win32 just for the shell commands it likes to execute.

The other option, assuming the cluster is just for local development, is 
have an ant build file to do the classpath setup. They have done the 
windows and cygwin support [1], and if you use <java> with 
failonerror=true and failonerror=false the ant process will stay hooked 
up (so streaming console output both ways), and pass up failures to the 

Just a thought.


[1] http://ant.apache.org/manual/running.html

View raw message