hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Guedes <ps-gue...@criticalsoftware.com>
Subject Re: Newbie question about running hadoop example
Date Mon, 14 May 2007 08:59:49 GMT
Hello,

Glad I could help...

I also didn't understand why hadoop uses hostnames even when IP's are 
specified in the configuration files... in my (perhaps misinformed) 
opinion hadoop should avoid name resolution has much has it can, because 
it's just another point of failure... Can anybody explain why this happens?

Pedro Guedes

Yiqing Jin wrote:
> thanks for your help Pedro,
>
> i edited my /etc/hosts and added all the nodes as you said, after that 
> the
> program works fine.
>
> i just don't get it why we need to connect using DNS since we already 
> have
> the ip address, and we only get connection failure at program run 
> time....
> that is hard for me to find out if without your help., thank you.
>
>
>
> 2007/5/11, Pedro Guedes <ps-guedes@criticalsoftware.com>:
>>
>> Hello,
>>
>> I had a similar problem with hadoop some time ago... by looking at the
>> logs, I found out that the problem was with resolving hostnames when the
>> HDFS nodes communicated with each other...
>> I fixed it by editing my /etc/hosts and adding all the nodes I'm 
>> using...
>>
>> Hope this helps,
>> Pedro Guedes
>>
>> Yiqing Jin wrote:
>> > Hi,
>> > I;m new to hadoop and i downloaded hadoop-0.12.3 from apahce 
>> website and
>> > installed it to a server with RedHat EL4. I followed the instruction
>> from
>> > wiki page:
>> > 1. formatted the namenod,.
>> > 2. use hadoop dfs -put to put a txt file to dfs(It's the 
>> LICENSE.txt in
>> > hadoop root directory).
>> > 3. use hadoop dfs -ls , and the file is listed correctly.
>> > 4. try  ./hadoop jar ../hadoop-0.12.3-examples.jar grep /user/hadoop/
>> > /user/hadoop/out apache,  everything works fine and i got the result.
>> >
>> > Next i tried to Starting up a larger cluster, set the ssh and scp a
>> > copy to
>> > another server, formatted the namenode again, started hadoop using
>> > start-all.sh.
>> > Then i tried the example again, but it seems will not finish and just
>> > hanging there:
>> >
>> > [hadoop@linux5 bin]$ ./start-all.sh
>> > starting namenode, logging to
>> > /home/hadoop/hadoop-0.12.3/bin/../logs/hadoop-
>> > hadoop-namenode-linux5.out
>> > localhost: starting datanode, logging to /home/hadoop/hadoop-0.12.3
>> > /bin/../logs/hadoop-hadoop-datanode-linux5.out
>> > 172.16.201.6: starting datanode, logging to /home/hadoop/hadoop-0.12.3
>> > /bin/../logs/hadoop-hadoop-datanode-linux6.out
>> > localhost: starting secondarynamenode, logging to
>> > /home/hadoop/hadoop-0.12.3
>> > /bin/../logs/hadoop-hadoop-secondarynamenode-linux5.out
>> > starting jobtracker, logging to /home/hadoop/hadoop-0.12.3
>> > /bin/../logs/hadoop-hadoop-jobtracker-linux5.out
>> > 172.16.201.6: starting tasktracker, logging to /home/hadoop/hadoop-
>> 0.12.3
>> > /bin/../logs/hadoop-hadoop-tasktracker-linux6.out
>> > localhost: starting tasktracker, logging to /home/hadoop/hadoop-0.12.3
>> > /bin/../logs/hadoop-hadoop-tasktracker-linux5.out
>> >
>> > [hadoop@linux5 bin]$ ./hadoop jar ../hadoop-0.12.3-examples.jar grep
>> > /user/hadoop/ /user/hadoop/out apache
>> > 07/05/11 18:34:49 INFO mapred.InputFormatBase: Total input paths to
>> > process
>> > : 1
>> > 07/05/11 18:34:49 INFO mapred.JobClient: Running job: job_0002
>> > 07/05/11 18:34:50 INFO mapred.JobClient:  map 0% reduce 0%
>> > 07/05/11 18:34:56 INFO mapred.JobClient:  map 50% reduce 0%
>> > 07/05/11 18:34:57 INFO mapred.JobClient:  map 100% reduce 0%
>> > 07/05/11 18:35:05 INFO mapred.JobClient:  map 100% reduce 8%
>> > 07/05/11 18:35:06 INFO mapred.JobClient:  map 100% reduce 16%
>> >
>> > the reduce task just processed to 16%.
>> > So i waited about an hour the job is still here, could some one help
>> > me find
>> > out what i did wrong?
>> >
>> > Thanks,
>> > Yiqing
>> >
>>
>>
>


Mime
View raw message