hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rasit OZDAS <rasitoz...@gmail.com>
Subject Re: Regarding "Hadoop multi cluster" set-up
Date Thu, 05 Feb 2009 15:19:27 GMT
Ian,
here is a list under
Setting up Hadoop on a single node > Basic Configuration > Jobtracker
and Namenode settings
Maybe it's what you're looking for.

Cheers,
Rasit

2009/2/4 Ian Soboroff <ian.soboroff@nist.gov>:
> I would love to see someplace a complete list of the ports that the various
> Hadoop daemons expect to have open.  Does anyone have that?
>
> Ian
>
> On Feb 4, 2009, at 1:16 PM, shefali pawar wrote:
>
>>
>> Hi,
>>
>> I will have to check. I can do that tomorrow in college. But if that is
>> the case what should i do?
>>
>> Should i change the port number and try again?
>>
>> Shefali
>>
>> On Wed, 04 Feb 2009 S D wrote :
>>>
>>> Shefali,
>>>
>>> Is your firewall blocking port 54310 on the master?
>>>
>>> John
>>>
>>> On Wed, Feb 4, 2009 at 12:34 PM, shefali pawar
>>> <shefali_p@rediffmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to set-up a two node cluster using Hadoop0.19.0, with 1
>>>> master(which should also work as a slave) and 1 slave node.
>>>>
>>>> But while running bin/start-dfs.sh the datanode is not starting on the
>>>> slave. I had read the previous mails on the list, but nothing seems to
>>>> be
>>>> working in this case. I am getting the following error in the
>>>> hadoop-root-datanode-slave log file while running the command
>>>> bin/start-dfs.sh =>
>>>>
>>>> 2009-02-03 13:00:27,516 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>>> /************************************************************
>>>> STARTUP_MSG: Starting DataNode
>>>> STARTUP_MSG:   host = slave/172.16.0.32
>>>> STARTUP_MSG:   args = []
>>>> STARTUP_MSG:   version = 0.19.0
>>>> STARTUP_MSG:   build =
>>>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.19 -r
>>>> 713890; compiled by 'ndaley' on Fri Nov 14 03:12:29 UTC 2008
>>>> ************************************************************/
>>>> 2009-02-03 13:00:28,725 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 0 time(s).
>>>> 2009-02-03 13:00:29,726 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 1 time(s).
>>>> 2009-02-03 13:00:30,727 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 2 time(s).
>>>> 2009-02-03 13:00:31,728 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 3 time(s).
>>>> 2009-02-03 13:00:32,729 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 4 time(s).
>>>> 2009-02-03 13:00:33,730 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 5 time(s).
>>>> 2009-02-03 13:00:34,731 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 6 time(s).
>>>> 2009-02-03 13:00:35,732 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 7 time(s).
>>>> 2009-02-03 13:00:36,733 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 8 time(s).
>>>> 2009-02-03 13:00:37,734 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/172.16.0.46:54310. Already tried 9 time(s).
>>>> 2009-02-03 13:00:37,738 ERROR
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>>> Call
>>>> to master/172.16.0.46:54310 failed on local exception: No route to host
>>>>      at org.apache.hadoop.ipc.Client.call(Client.java:699)
>>>>      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>>>>      at $Proxy4.getProtocolVersion(Unknown Source)
>>>>      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319)
>>>>      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:306)
>>>>      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:343)
>>>>      at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:288)
>>>>      at
>>>>
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:258)
>>>>      at
>>>>
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
>>>>      at
>>>>
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1199)
>>>>      at
>>>>
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1154)
>>>>      at
>>>>
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1162)
>>>>      at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1284)
>>>> Caused by: java.net.NoRouteToHostException: No route to host
>>>>      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>      at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>>>      at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
>>>>      at
>>>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
>>>>      at
>>>> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
>>>>      at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
>>>>      at org.apache.hadoop.ipc.Client.call(Client.java:685)
>>>>      ... 12 more
>>>>
>>>> 2009-02-03 13:00:37,739 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>>> /************************************************************
>>>> SHUTDOWN_MSG: Shutting down DataNode at slave/172.16.0.32
>>>> ************************************************************/
>>>>
>>>>
>>>> Also, the Pseudo distributed operation is working on both the machines.
>>>> And
>>>> i am able to ssh from 'master to master' and 'master to slave' via a
>>>> password-less ssh login. I do not think there is any problem with the
>>>> network because cross pinging is working fine.
>>>>
>>>> I am working on Linux (Fedora 8)
>>>>
>>>> The following is the configuration which i am using
>>>>
>>>> On master and slave, <HADOOP_INSTALL>/conf/masters looks like this:
>>>>
>>>> master
>>>>
>>>> On master and slave, <HADOOP_INSTALL>/conf/slaves looks like this:
>>>>
>>>> master
>>>> slave
>>>>
>>>> On both the machines conf/hadoop-site.xml looks like this
>>>>
>>>> <property>
>>>>  <name>fs.default.name</name>
>>>>  <value>hdfs://master:54310</value>
>>>>  <description>The name of the default file system.  A URI whose
>>>>  scheme and authority determine the FileSystem implementation.  The
>>>>  uri's scheme determines the config property (fs.SCHEME.impl) naming
>>>>  the FileSystem implementation class.  The uri's authority is used to
>>>>  determine the host, port, etc. for a filesystem.</description>
>>>> </property>
>>>> <property>
>>>>  <name>mapred.job.tracker</name>
>>>>  <value>master:54311</value>
>>>>  <description>The host and port that the MapReduce job tracker runs
>>>>  at.  If "local", then jobs are run in-process as a single map
>>>>  and reduce task.
>>>>  </description>
>>>> </property>
>>>> <property>
>>>>  <name>dfs.replication</name>
>>>>  <value>2</value>
>>>>  <description>Default block replication.
>>>>  The actual number of replications can be specified when the file is
>>>> created.
>>>>  The default is used if replication is not specified in create time.
>>>>  </description>
>>>> </property>
>>>>
>>>> namenode is formatted succesfully by running
>>>>
>>>> "bin/hadoop namenode -format"
>>>>
>>>> on the master node.
>>>>
>>>> I am new to Hadoop and I do not know what is going wrong.
>>>>
>>>> Any help will be appreciated.
>>>>
>>>> Thanking you in advance
>>>>
>>>> Shefali Pawar
>>>> Pune, India
>>>>
>
>



-- 
M. Raşit ÖZDAŞ

Mime
View raw message