Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 96242 invoked from network); 26 Jul 2008 00:41:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 26 Jul 2008 00:41:23 -0000 Received: (qmail 69837 invoked by uid 500); 26 Jul 2008 00:41:18 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 69795 invoked by uid 500); 26 Jul 2008 00:41:18 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 69784 invoked by uid 99); 26 Jul 2008 00:41:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jul 2008 17:41:18 -0700 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jmvidal@gmail.com designates 64.233.184.229 as permitted sender) Received: from [64.233.184.229] (HELO wr-out-0506.google.com) (64.233.184.229) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jul 2008 00:40:21 +0000 Received: by wr-out-0506.google.com with SMTP id c30so3332095wra.21 for ; Fri, 25 Jul 2008 17:40:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=y/N6pRzc8lOuErxa9SiObjFDfKlWNUFBVl5DxHmfFO8=; b=qxzBynOssUYciGGYuIFojhSAil2jiHu8LE2qRAG5hR4MYWyS/3FHzD0HaKu2FFRtg5 oOPP7oMEIoDdN7o1IMXxJ3RHRA+ZlwimYq3c2O6HWAWiMVeE5ey8fzyg1toBZ5loWsnZ rUl0nCjrRzR+7JYGbVrAtpzJgLTcQ1kyf/tl0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=TkSbr9Vz9RmsWvBKCEhsLUqibTDeaRq6pNT6JmfaedTOiLIubhW4X8+mnYaWaTMsHc Z6YJzbBqx8WtCnRwSagFpKjvwMon5K0Sfz3LjiYWv/ODkammXYlQaH+0OVuutlZtSevC 7DmKOPrXwCBzNiR9RQMpVnFqc4NBPCUuOnS4o= Received: by 10.90.78.10 with SMTP id a10mr2996403agb.38.1217032828613; Fri, 25 Jul 2008 17:40:28 -0700 (PDT) Received: by 10.90.89.17 with HTTP; Fri, 25 Jul 2008 17:40:28 -0700 (PDT) Message-ID: <1fbc5b4a0807251740h7fd50f4dkc212e7907d8d73bf@mail.gmail.com> Date: Fri, 25 Jul 2008 20:40:28 -0400 From: "Jose Vidal" To: core-user@hadoop.apache.org Subject: Re: newbie install In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <1fbc5b4a0807221403r63f3b875n2974b202912e97d6@mail.gmail.com> <1fbc5b4a0807221545x6e955c0bk11a692e8c3133053@mail.gmail.com> <1fbc5b4a0807230705l7def3f52q6963dc32da192f42@mail.gmail.com> X-Virus-Checked: Checked by ClamAV on apache.org Turns out, it does cause problems later on. I think the problem is that the slaves have, in their hosts files: 127.0.0.1 localhost.localdomain localhost 127.0.0.1 machinename.cse.sc.edu machinename The reduce phase fails because the reducer cannot get data from the mappers as it tries to open a connection to "http://localhost:...." This is kinda annoying as all the hostnames resolve properly using DNS. I think it qualify as a hadoop bug, or maybe not. Jose On Wed, Jul 23, 2008 at 10:19 AM, Edward J. Yoon wrote: > That's good. :) > >> Will this cause bigger problems later on? or should I just ignore it. > > I'm not sure, But I guess there is no problem. > Does anyone have some experience with that? > > Regards, Edward J. Yoon > > On Wed, Jul 23, 2008 at 11:05 PM, Jose Vidal wrote: >> Thanks! that worked. I was able to run dfs and put some files in it. >> >> However, when I go to my namenode at http://namenode:50070 I see that >> all the datanodes have a name of "localhost". >> >> Will this cause bigger problems later on? or should I just ignore it. >> >> Jose >> >> On Tue, Jul 22, 2008 at 6:48 PM, Edward J. Yoon wrote: >>>> So, do I need to change the host file in all the slaves, or just the namenode? >>> >>> Just the namenode. >>> >>> Thanks, Edward >>> >>> On Wed, Jul 23, 2008 at 7:45 AM, Jose Vidal wrote: >>>> Yes, the host file just has: >>>> >>>> 127.0.0.1 localhost hermes.cse.sc.edu hermes >>>> >>>> So, do I need to change the host file in all the slaves, or just the namenode? >>>> >>>> I'm not root on these machines so changing these requires gentle >>>> handling of our sysadmin.... >>>> >>>> Jose >>>> >>>> On Tue, Jul 22, 2008 at 5:37 PM, Edward J. Yoon wrote: >>>>> If you have a static address for the machine, make sure that your >>>>> hosts file is pointing to the static address for the namenode host >>>>> name as opposed to the 127.0.0.1 address. It should look something >>>>> like this with the values replaced with your values. >>>>> >>>>> 127.0.0.1 localhost.localdomain localhost >>>>> 192.x.x.x yourhost.yourdomain.com yourhost >>>>> >>>>> - Edward >>>>> >>>>> On Wed, Jul 23, 2008 at 6:03 AM, Jose Vidal wrote: >>>>>> I'm trying to install hadoop on our linux machine but after >>>>>> start-all.sh none of the slaves can connect: >>>>>> >>>>>> 2008-07-22 16:35:27,534 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG: >>>>>> /************************************************************ >>>>>> STARTUP_MSG: Starting DataNode >>>>>> STARTUP_MSG: host = thetis/127.0.0.1 >>>>>> STARTUP_MSG: args = [] >>>>>> STARTUP_MSG: version = 0.16.4 >>>>>> STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/core/branches/bran >>>>>> ch-0.16 -r 652614; compiled by 'hadoopqa' on Fri May 2 00:18:12 UTC 2008 >>>>>> ************************************************************/ >>>>>> 2008-07-22 16:35:27,643 WARN org.apache.hadoop.dfs.DataNode: Invalid directory i >>>>>> n dfs.data.dir: directory is not writable: /work >>>>>> 2008-07-22 16:35:27,699 INFO org.apache.hadoop.ipc.Client: Retrying connect to s >>>>>> erver: hermes.cse.sc.edu/129.252.130.148:9000. Already tried 1 time(s). >>>>>> 2008-07-22 16:35:28,700 INFO org.apache.hadoop.ipc.Client: Retrying connect to s >>>>>> erver: hermes.cse.sc.edu/129.252.130.148:9000. Already tried 2 time(s). >>>>>> 2008-07-22 16:35:29,700 INFO org.apache.hadoop.ipc.Client: Retrying connect to s >>>>>> erver: hermes.cse.sc.edu/129.252.130.148:9000. Already tried 3 time(s). >>>>>> 2008-07-22 16:35:30,701 INFO org.apache.hadoop.ipc.Client: Retrying connect to s >>>>>> erver: hermes.cse.sc.edu/129.252.130.148:9000. Already tried 4 time(s). >>>>>> 2008-07-22 16:35:31,702 INFO org.apache.hadoop.ipc.Client: Retrying connect to s >>>>>> erver: hermes.cse.sc.edu/129.252.130.148:9000. Already tried 5 time(s). >>>>>> 2008-07-22 16:35:32,702 INFO org.apache.hadoop.ipc.Client: Retrying connect to s >>>>>> erver: hermes.cse.sc.edu/129.252.130.148:9000. Already tried 6 time(s). >>>>>> >>>>>> same for the tasktrackers (port 9001). >>>>>> >>>>>> I think the problem has something to do with name resolution. Check these out: >>>>>> >>>>>> jmvidal@hermes:~/hadoop-0.16.4> telnet hermes.cse.sc.edu 9000 >>>>>> Trying 127.0.0.1... >>>>>> Connected to hermes.cse.sc.edu (127.0.0.1). >>>>>> Escape character is '^]'. >>>>>> bye >>>>>> Connection closed by foreign host. >>>>>> >>>>>> jmvidal@hermes:~/hadoop-0.16.4> host hermes.cse.sc.edu >>>>>> hermes.cse.sc.edu has address 129.252.130.148 >>>>>> >>>>>> jmvidal@hermes:~/hadoop-0.16.4> telnet 129.252.130.148 9000 >>>>>> Trying 129.252.130.148... >>>>>> telnet: connect to address 129.252.130.148: Connection refused >>>>>> telnet: Unable to connect to remote host: Connection refused >>>>>> >>>>>> So, the first one connects but not the second one, but they both go to >>>>>> the same machine:port. My guess is that the hadoop server is closing >>>>>> the connection, but why? >>>>>> >>>>>> Thanks, >>>>>> Jose >>>>>> >>>>>> -- >>>>>> Jose M. Vidal http://jmvidal.cse.sc.edu >>>>>> University of South Carolina http://www.multiagent.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Edward J. Yoon, >>>>> http://blog.udanax.org >>>>> >>>> >>>> >>>> >>>> -- >>>> Jose M. Vidal http://jmvidal.cse.sc.edu >>>> University of South Carolina http://www.multiagent.com >>>> >>> >>> >>> >>> -- >>> Best regards, Edward J. Yoon >>> edwardyoon@apache.org >>> http://blog.udanax.org >>> >> >> >> >> -- >> Jose M. Vidal http://jmvidal.cse.sc.edu >> University of South Carolina http://www.multiagent.com >> > > > > -- > Best regards, Edward J. Yoon > edwardyoon@apache.org > http://blog.udanax.org > -- Jose M. Vidal http://jmvidal.cse.sc.edu University of South Carolina http://www.multiagent.com