hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Lothian" <nloth...@educationau.edu.au>
Subject Hadoop on Windows
Date Mon, 08 Oct 2007 05:43:57 GMT


I'm trying to get Hadoop running on Windows, and I've found that it
isn't exactly a simple process (yes, I'm using Cygwin).

I've had a quick look though the mailing list, and I see the occasional
question about it, generally with the response "use Cygwin", and no
further details.

Is anyone actually successfully running Hadoop on Windows/Cygwin? If so,
have you had similar problems to those outlined below? I think there's a
fair chance the first two of these are environmental local to my
machine, but it would be nice to be sure that if I keep working on this
it will work in the end...

1) Running bin/start-all.sh gives:
: no address associated with name

That turns out to be a Unix vs DOS line endings thing (!). 

Running the following commands fixed that:

dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters
dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves

2) Any script using hadoop-daemon.sh failed. This turned out to be
another Unix vs DOS line endings thing. The critical error looked like

<machine-name>: head: cannot open
-<machine-name>\r.out' for reading: No such file or directory

(note that "\r.out" in the logfile name...)

The fix for that was to alter the hadoop-daemon.sh script. Replace:

HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' | sed


log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' |
sed s/\r//`.out

3) Now the start-all.sh script seems to run ok, but the Java processes
which are started on the slave machine (in this case the same machine as
the master) die soon after launching.

If I manually run:
bin/hadoop jobtracker
bin/hadoop tasktracker
bin/hadoop namenode
bin/hadoop datanode

I can succesfully submit jobs. However, when using the start-all.sh
script the submitted jobs fail to connect (because the jobtracker dies).
Has anyone else seen this? Is there a workaround?

  Nick Lothian

IMPORTANT: This e-mail, including any attachments, may contain private or confidential information.
If you think you may not be the intended recipient, or if you have received this e-mail in
error, please contact the sender immediately and delete all copies of this e-mail. 
If you are not the intended recipient, you must not reproduce any part of this e-mail or disclose
its contents to any other party. 
This email represents the views of the individual sender, which do not necessarily reflect
those of education.au limited except where the sender expressly states otherwise. 
It is your responsibility to scan this email and any files transmitted with it for viruses
or any other defects. education.au limited will not be liable for any loss, damage or consequence
caused directly or indirectly by this email.

View raw message