From hadoop-user-return-2478-apmail-lucene-hadoop-user-archive=lucene.apache.org@lucene.apache.org Mon Oct 08 05:44:55 2007 Return-Path: Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org Received: (qmail 28177 invoked from network); 8 Oct 2007 05:44:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 8 Oct 2007 05:44:54 -0000 Received: (qmail 64880 invoked by uid 500); 8 Oct 2007 05:44:41 -0000 Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org Received: (qmail 64847 invoked by uid 500); 8 Oct 2007 05:44:41 -0000 Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-user@lucene.apache.org Delivered-To: mailing list hadoop-user@lucene.apache.org Received: (qmail 64838 invoked by uid 99); 8 Oct 2007 05:44:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Oct 2007 22:44:41 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nlothian@educationau.edu.au designates 203.9.156.1 as permitted sender) Received: from [203.9.156.1] (HELO mail.educationau.edu.au) (203.9.156.1) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 08 Oct 2007 05:44:44 +0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.3959 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Hadoop on Windows Date: Mon, 8 Oct 2007 15:13:57 +0930 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Hadoop on Windows thread-index: AcgJbjiF2QAnGq1+R4qaXt1c80wiQg== From: "Nick Lothian" To: X-Virus-Checked: Checked by ClamAV on apache.org Hi, I'm trying to get Hadoop running on Windows, and I've found that it isn't exactly a simple process (yes, I'm using Cygwin). I've had a quick look though the mailing list, and I see the occasional question about it, generally with the response "use Cygwin", and no further details. Is anyone actually successfully running Hadoop on Windows/Cygwin? If so, have you had similar problems to those outlined below? I think there's a fair chance the first two of these are environmental local to my machine, but it would be nice to be sure that if I keep working on this it will work in the end... 1) Running bin/start-all.sh gives: : no address associated with name That turns out to be a Unix vs DOS line endings thing (!).=20 Running the following commands fixed that: dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves 2) Any script using hadoop-daemon.sh failed. This turned out to be another Unix vs DOS line endings thing. The critical error looked like this: : head: cannot open `/cygdrive/c/dev/prog/hadoop-0.14.1/bin/../logs/hadoop-nlothian-namenode -\r.out' for reading: No such file or directory (note that "\r.out" in the logfile name...) The fix for that was to alter the hadoop-daemon.sh script. Replace: export HADOOP_LOGFILE=3Dhadoop-$HADOOP_IDENT_STRING-$command-`hostname`.log=20 with=20 export HADOOP_LOGFILE=3Dhadoop-$HADOOP_IDENT_STRING-$command-`'hostname' | sed s/\r//`.log and=20 =09 log=3D$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.out= =20 with=20 =09 log=3D$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' | sed s/\r//`.out 3) Now the start-all.sh script seems to run ok, but the Java processes which are started on the slave machine (in this case the same machine as the master) die soon after launching. If I manually run: bin/hadoop jobtracker bin/hadoop tasktracker bin/hadoop namenode bin/hadoop datanode I can succesfully submit jobs. However, when using the start-all.sh script the submitted jobs fail to connect (because the jobtracker dies). Has anyone else seen this? Is there a workaround? Regards Nick Lothian IMPORTANT: This e-mail, including any attachments, may contain private = or confidential information. If you think you may not be the intended recipient, or if you have = received this e-mail in error, please contact the sender immediately and = delete all copies of this e-mail.=20 If you are not the intended recipient, you must not reproduce any part = of this e-mail or disclose its contents to any other party.=20 This email represents the views of the individual sender, which do not = necessarily reflect those of education.au limited except where the = sender expressly states otherwise.=20 It is your responsibility to scan this email and any files transmitted = with it for viruses or any other defects. education.au limited will not = be liable for any loss, damage or consequence caused directly or = indirectly by this email.