hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shanmuganathan.r" <shanmuganatha...@zohocorp.com>
Subject Re: hadoop cluster mode not starting up
Date Wed, 17 Aug 2011 12:13:43 GMT
Hi Df,

      check that you have w1153435 user in all machines in the cluster  




and use the same configuration for all machines. use IP instead of name. (already you said
that you didn't have the root permission  )



&lt;name&gt;fs.default.name&lt;/name&gt;
&lt;value&gt;hdfs://109.9.3.101(ex):3000&lt;/value&gt;


&lt;name&gt;mapred.job.tracker&lt;/name&gt;
&lt;value&gt;109.9.3.101(ex):3001&lt;/value&gt;


      check that ssh passwordless login

Regards,

Shanmuganathan



---- On Wed, 17 Aug 2011 17:12:25 +0530 A Df &lt;abbey_dragonforest@yahoo.com&gt;
wrote ---- 


Hello Everyone:

I am adding the contents of my config file in the hopes that someone will be able to help.
See inline for the discussions. I really don't understand why it works in pseudo-mode but
gives so much problems in cluster. I have tried the instructions from the Apache cluster setup,
Yahoo Development Network and from Michael Noll's tutorial.

w1153435@ngs:~/hadoop-0.20.2_cluster/conf&gt; cat core-site.xml
&lt;?xml version="1.0"?&gt;
&lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;

&lt;!-- Put site-specific property overrides in this file. --&gt;


&lt;configuration&gt;
     &lt;property&gt;
         &lt;name&gt;fs.default.name&lt;/name&gt;
         &lt;value&gt;hdfs://ngs.uni.ac.uk:3000&lt;/value&gt;
     &lt;/property&gt;
     &lt;property&gt;
&lt;name&gt;HADOOP_LOG_DIR&lt;/name&gt;
         &lt;value&gt;/home/w1153435/hadoop-0.20.2_cluster/var/log/hadoop&lt;/value&gt;
     &lt;/property&gt;
 &lt;property&gt;
&lt;name&gt;hadoop.tmp.dir&lt;/name&gt;
         &lt;value&gt;/home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop&lt;/value&gt;
     &lt;/property&gt;
&lt;/configuration&gt;

w1153435@ngs:~/hadoop-0.20.2_cluster/conf&gt; cat hdfs-site.xml
&lt;?xml version="1.0"?&gt;
&lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;

&lt;!-- Put site-specific property overrides in this file. --&gt;

&lt;configuration&gt;
     &lt;property&gt;
         &lt;name&gt;dfs.replication&lt;/name&gt;
         &lt;value&gt;3&lt;/value&gt;
     &lt;/property&gt;
         &lt;property&gt;
         &lt;name&gt;dfs.http.address&lt;/name&gt;
         &lt;value&gt;0.0.0.0:3500&lt;/value&gt;
     &lt;/property&gt;
&lt;property&gt;
    &lt;name&gt;dfs.data.dir&lt;/name&gt;
    &lt;value&gt;/home/w1153435/hadoop-0.20.2_cluster/dfs/data&lt;/value&gt;
    &lt;final&gt;true&lt;/final&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.name.dir&lt;/name&gt;
    &lt;value&gt;/home/w1153435/hadoop-0.20.2_cluster/dfs/name&lt;/value&gt;
    &lt;final&gt;true&lt;/final&gt;
  &lt;/property&gt;
&lt;/configuration&gt;

w1153435@ngs:~/hadoop-0.20.2_cluster/conf&gt; cat mapred-site.xml
&lt;?xml version="1.0"?&gt;
&lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;

&lt;!-- Put site-specific property overrides in this file. --&gt;

&lt;configuration&gt;
     &lt;property&gt;
         &lt;name&gt;mapred.job.tracker&lt;/name&gt;
         &lt;value&gt;ngs.uni.ac.uk:3001&lt;/value&gt;
     &lt;/property&gt;
&lt;property&gt;
         &lt;name&gt;mapred.system.dir&lt;/name&gt;
         &lt;value&gt;/home/w1153435/hadoop-0.20.2_cluster/mapred/system&lt;/value&gt;
     &lt;/property&gt;
&lt;property&gt;
         &lt;name&gt;mapred.map.tasks&lt;/name&gt;
         &lt;value&gt;80&lt;/value&gt;
     &lt;/property&gt;
&lt;property&gt;
         &lt;name&gt;mapred.reduce.tasks&lt;/name&gt;
         &lt;value&gt;16&lt;/value&gt;
     &lt;/property&gt;

&lt;/configuration&gt;

In addition:

w1153435@ngs:~/hadoop-0.20.2_cluster&gt; bin/hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: �%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 161.74.12.36:50010
Decommission Status : Normal
Configured Capacity: 0 (0 KB)
DFS Used: 0 (0 KB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 0(0 KB)
DFS Used%: 100%
DFS Remaining%: 0%
Last contact: Wed Aug 17 12:40:17 BST 2011

Cheers,
A Df

&gt;________________________________
&gt;From: A Df &lt;abbey_dragonforest@yahoo.com&gt;
&gt;To: "common-user@hadoop.apache.org" &lt;common-user@hadoop.apache.org&gt;;
"shanmuganathan.r@zohocorp.com" &lt;shanmuganathan.r@zohocorp.com&gt;
&gt;Sent: Tuesday, 16 August 2011, 16:20
&gt;Subject: Re: hadoop cluster mode not starting up
&gt;
&gt;
&gt;
&gt;See inline:
&gt;
&gt;
&gt;&gt;________________________________
&gt;&gt;From: shanmuganathan.r &lt;shanmuganathan.r@zohocorp.com&gt;
&gt;&gt;To: common-user@hadoop.apache.org
&gt;&gt;Sent: Tuesday, 16 August 2011, 13:35
&gt;&gt;Subject: Re: hadoop cluster mode not starting up
&gt;&gt;
&gt;&gt;Hi Df,      
&gt;&gt;
&gt;&gt;      Are you use the IP instead of names in conf/masters and conf/slaves
. For running the secondary namenode in separate machine refer the following link 
&gt;&gt;
&gt;&gt;
&gt;&gt;=Yes, I use the names in those files but the ip address are mapped to the
names in the /extras/hosts file. Does this cause problems?
&gt;&gt;
&gt;&gt;
&gt;&gt;http://www.hadoop-blog.com/2010/12/secondarynamenode-process-is-starting.html

&gt;&gt;
&gt;&gt;
&gt;&gt;=I want to making too many changes so I will stick to having the master be
both namenode and secondarynamenode. I tried starting up the hdfs and mapreduce but the jobtracker
is not running on the master and their is still errors regarding the datanodes because only
5 of 7 datanodes have tasktracker. I ran both commands for to start the hdfs and mapreduce
so why is the jobtracker missing?
&gt;&gt;
&gt;&gt;Regards,
&gt;&gt;
&gt;&gt;Shanmuganathan
&gt;&gt;
&gt;&gt;
&gt;&gt;
&gt;&gt;---- On Tue, 16 Aug 2011 17:06:04 +0530 A Df&amp;lt;abbey_dragonforest@yahoo.com&amp;gt;
wrote ---- 
&gt;&gt;
&gt;&gt;
&gt;&gt;I already used a few tutorials as follows:
&gt;&gt;    * Hadoop Tutorial on Yahoo Developer network which uses an old hadoop
and thus older conf files.
&gt;&gt;
&gt;&gt;    * http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
which only has two nodes and the master acts as namenode and secondary namenode. I need one
with more than that.  
&gt;&gt;
&gt;&gt;
&gt;&gt;Is there a way to prevent the node from using the central file system because
I don't have root permission and my user folder is in a central file system which is replicated
on all the nodes?
&gt;&gt;
&gt;&gt;See inline too for my responses
&gt;&gt;
&gt;&gt;
&gt;&gt;
&gt;&gt;&amp;gt;________________________________
&gt;&gt;&amp;gt;From: Steve Loughran &amp;lt;stevel@apache.org&amp;gt;
&gt;&gt;&amp;gt;To: common-user@hadoop.apache.org
&gt;&gt;&amp;gt;Sent: Tuesday, 16 August 2011, 12:08
&gt;&gt;&amp;gt;Subject: Re: hadoop cluster mode not starting up
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;On 16/08/11 11:19, A Df wrote:
&gt;&gt;&amp;gt;&amp;gt; See inline
&gt;&gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; ________________________________
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; From: Steve Loughran&amp;lt;stevel@apache.org&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; To: common-user@hadoop.apache.org
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; Sent: Tuesday, 16 August 2011, 11:08
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; Subject: Re: hadoop cluster mode not
starting up
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; On 16/08/11 11:02, A Df wrote:
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; Hello All:
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; I used a combination of tutorials
to setup hadoop but most seems to be using either an old version of hadoop or only using 2
machines for the cluster which isn't really a cluster. Does anyone know of a good tutorial
which setups multiple nodes for a cluster?? I already looked at the Apache website but it
does not give sample values for the conf files. Also each set of tutorials seem to have a
different set of parameters which they indicate should be changed so now its a bit confusing.
For example, my configuration sets a dedicate namenode, secondary namenode and 8 slave nodes
but when I run the start command it gives an error. Should I install hadoop to my user directory
or on the root? I have it in my directory but all the nodes have a central file system as
opposed to distributed so whatever I do on one node in my user folder it affect all the others
so how do i set the paths to ensure that it uses a distributed system?
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; For the errors below, I checked
the directories and the files are there. Am I not sure what went wrong and how to set the
conf to not have central file system. Thank you.
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; Error message
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; CODE
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; w1153435@n51:~/hadoop-0.20.2_cluster&amp;gt;
 bin/start-dfs.sh
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; bin/start-dfs.sh: line 28:
/w1153435/hadoop-0.20.2_cluster/bin/hadoop-config.sh: No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; bin/start-dfs.sh: line 50:
/w1153435/hadoop-0.20.2_cluster/bin/hadoop-daemon.sh: No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; bin/start-dfs.sh: line 51:
/w1153435/hadoop-0.20.2_cluster/bin/hadoop-daemons.sh: No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; bin/start-dfs.sh: line 52:
/w1153435/hadoop-0.20.2_cluster/bin/hadoop-daemons.sh: No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; CODE
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; there's  No such file or directory as
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; /w1153435/hadoop-0.20.2_cluster/bin/hadoop-daemons.sh
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; There is, I checked as shown
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; w1153435@n51:~/hadoop-0.20.2_cluster&amp;gt;
 ls bin
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; hadoop            rcc               
start-dfs.sh      stop-dfs.sh
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; hadoop-config.sh  slaves.sh         
start-mapred.sh  stop-mapred.sh
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; hadoop-daemon.sh  start-all.sh      stop-all.sh
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; hadoop-daemons.sh  start-balancer.sh
 stop-balancer.sh
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;try "pwd" to print out where the OS thinks you are, as it doesn't
seem 
&gt;&gt;&amp;gt;to be where you think you are
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt; pwd
&gt;&gt;&amp;gt;/home/w1153435/hadoop-0.20.2_cluster
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;w1153435@ngs:~/hadoop-0.20.2_cluster/bin&amp;gt; pwd
&gt;&gt;&amp;gt;/home/w1153435/hadoop-0.20.2_cluster/bin
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; I had tried running this
command below earlier but also got problems:
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; CODE
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt;
 export HADOOP_CONF_DIR=${HADOOP_HOME}/conf
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt;
 export HADOOP_SLAVES=${HADOOP_CONF_DIR}/slaves
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt;
 ${HADOOP_HOME}/bin/slaves.sh "mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop"
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; -bash: /bin/slaves.sh: No
such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt;
 export HADOOP_HOME=/home/w1153435/hadoop-0.20.2_cluster
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt;
 ${HADOOP_HOME}/bin/slaves.sh "mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop"
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; cat: /conf/slaves: No such
file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; CODE
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; there's  No such file or directory as
/conf/slaves because you set
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; HADOOP_HOME after setting the other env
variables, which are expanded at
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; set-time, not run-time.
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; I redid the command but still have errors
on the slaves
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt;
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; w1153435@n51:~/hadoop-0.20.2_cluster&amp;gt;
 export HADOOP_HOME=/home/w1153435/hadoop-0.20.2_cluster
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; w1153435@n51:~/hadoop-0.20.2_cluster&amp;gt;
 export HADOOP_CONF_DIR=${HADOOP_HOME}/conf
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; w1153435@n51:~/hadoop-0.20.2_cluster&amp;gt;
 export HADOOP_SLAVES=${HADOOP_CONF_DIR}/slaves
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; w1153435@n51:~/hadoop-0.20.2_cluster&amp;gt;
 ${HADOOP_HOME}/bin/slaves.sh "mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop"
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; privn51: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; privn58: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; privn52: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; privn55: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; privn57: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; privn54: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; privn53: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&amp;gt;&amp;gt;&amp;gt; privn56: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;try ssh-ing in, do it by hand, make sure you have the right permissions
etc
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;I reset the above path variables again and checked that they existed
and tried the command above but same error. I used ssh with no problems and no password request
so that is fine. What else could be wrong?
&gt;&gt;&amp;gt;w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt; echo $HADOOP_HOME
                        /home/w1153435/hadoop-0.20.2_cluster
&gt;&gt;&amp;gt;w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt; echo $HADOOP_CONF_DIR
                    /home/w1153435/hadoop-0.20.2_cluster/conf
&gt;&gt;&amp;gt;w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt; echo $HADOOP_SLAVES
                      /home/w1153435/hadoop-0.20.2_cluster/conf/slaves
&gt;&gt;&amp;gt;w1153435@ngs:~/hadoop-0.20.2_cluster&amp;gt;
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;
&gt;&gt;&amp;gt;
&gt;&gt;
&gt;&gt;
&gt;&gt;
&gt;&gt;
&gt;
&gt;


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message