I'm able to setup hadoop clusters with required configurations. I can see that all the required services on master and on slaves nodes are running as required[please see below JPS command output ]. The problem, I'm facing is that, the HDFS and Mapreduce daemons running on Master and can be accessed from Master only, and not from the slave machines. Note that, I've added these ports in the EC2 security group to open them. And I can browse the master machines UI from web browser, using: http://<machine ip>:50070/dfshealth.jsp
Now, the problem which I'm facing is , the HDFS as well the jobtracker both are accessible from the master machine[I'm using master as both Namenode and Datanode] but both the ports[hdfs: 54310 and mapreduce: 54320] used for these two are not accessible from other slave nodes.
I did: netstat -puntl on master machine and got this:
hadoop@nutchcluster1:~/hadoop$ netstat -puntl
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::50020 :::* LISTEN 6224/java
tcp6 0 0 127.0.0.1:54310 :::* LISTEN 6040/java
tcp6 0 0 127.0.0.1:32776 :::* LISTEN 6723/java
tcp6 0 0 :::57065 :::* LISTEN 6040/java
tcp6 0 0 :::50090 :::* LISTEN 6401/java
tcp6 0 0 :::50060 :::* LISTEN 6723/java
tcp6 0 0 :::50030 :::* LISTEN 6540/java
tcp6 0 0 127.0.0.1:54320 :::* LISTEN 6540/java
tcp6 0 0 :::45747 :::* LISTEN 6401/java
tcp6 0 0 :::33174 :::* LISTEN 6540/java
tcp6 0 0 :::50070 :::* LISTEN 6040/java
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::54424 :::* LISTEN 6224/java
tcp6 0 0 :::50010 :::* LISTEN 6224/java
tcp6 0 0 :::50075 :::* LISTEN 6224/java
udp 0 0 0.0.0.0:68 0.0.0.0:* -
As can be seen in the output, both the HDFS daemon and mapreduce daemons are accessible, but only from 127.0.0.1 and not from 0.0.0.0 [any machine/slave machines]
<description>A base for other temporary directories</description>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
-----------------hdfs-site.xml content is as below:
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
------------------mapred-site.xml content is as:
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
<description>As a rule of thumb, use 10x the number of slaves (i.e., number of tasktrackers).
<description>As a rule of thumb, use 2x the number of slave processors (i.e., number of tasktrackers).
I replicated all the above on all the other 3 slave machines[1 master + 3 slaves]. My /etc/hosts content is as below on the master node. Note that, I've thee same content on slaves as well, the only difference is, its own IP is set to 127.0.0.1 and for others the exact IP is set:
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
-----------------/etc/hosts content ends here
File content for masters is :
and file content for slaves is:
Then, I copied all the contents relevant to config[*-site.xml, *.env files] folder on all the slaves.
As per the steps, I'm starting the HDFS using: bin/start-dfs.sh and then I'm starting the mapreduce as : bin/start-mapred.sh
After running the above two on my master machine[nuthccluster1] I can see the following jps output:
and on the slaves, the jps output is:
This clearly indicates that the port 54310 is accessible from the master only and not from the slaves. This is the point I'm stuck at and would appreciate if someone could point me what config is missing or what is wrong. Any comment/feedback, in this regard would be highly appreciated. Thanks in advance.