hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shanmuganathan.r" <shanmuganatha...@zohocorp.com>
Subject Re: hadoop cluster mode not starting up
Date Tue, 16 Aug 2011 12:35:23 GMT
Hi Df,      

      Are you use the IP instead of names in conf/masters and conf/slaves . For running the
secondary namenode in separate machine refer the following link 



http://www.hadoop-blog.com/2010/12/secondarynamenode-process-is-starting.html 


Regards,

Shanmuganathan



---- On Tue, 16 Aug 2011 17:06:04 +0530 A Df&lt;abbey_dragonforest@yahoo.com&gt; wrote
---- 


I already used a few tutorials as follows:
    * Hadoop Tutorial on Yahoo Developer network which uses an old hadoop and thus older conf
files.

    * http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
which only has two nodes and the master acts as namenode and secondary namenode. I need one
with more than that.  


Is there a way to prevent the node from using the central file system because I don't have
root permission and my user folder is in a central file system which is replicated on all
the nodes?

See inline too for my responses



&gt;________________________________
&gt;From: Steve Loughran &lt;stevel@apache.org&gt;
&gt;To: common-user@hadoop.apache.org
&gt;Sent: Tuesday, 16 August 2011, 12:08
&gt;Subject: Re: hadoop cluster mode not starting up
&gt;
&gt;On 16/08/11 11:19, A Df wrote:
&gt;&gt; See inline
&gt;&gt;
&gt;&gt;
&gt;&gt;
&gt;&gt;&gt; ________________________________
&gt;&gt;&gt; From: Steve Loughran&lt;stevel@apache.org&gt;
&gt;&gt;&gt; To: common-user@hadoop.apache.org
&gt;&gt;&gt; Sent: Tuesday, 16 August 2011, 11:08
&gt;&gt;&gt; Subject: Re: hadoop cluster mode not starting up
&gt;&gt;&gt;
&gt;&gt;&gt; On 16/08/11 11:02, A Df wrote:
&gt;&gt;&gt;&gt; Hello All:
&gt;&gt;&gt;&gt;
&gt;&gt;&gt;&gt; I used a combination of tutorials to setup hadoop but most
seems to be using either an old version of hadoop or only using 2 machines for the cluster
which isn't really a cluster. Does anyone know of a good tutorial which setups multiple nodes
for a cluster?? I already looked at the Apache website but it does not give sample values
for the conf files. Also each set of tutorials seem to have a different set of parameters
which they indicate should be changed so now its a bit confusing. For example, my configuration
sets a dedicate namenode, secondary namenode and 8 slave nodes but when I run the start command
it gives an error. Should I install hadoop to my user directory or on the root? I have it
in my directory but all the nodes have a central file system as opposed to distributed so
whatever I do on one node in my user folder it affect all the others so how do i set the paths
to ensure that it uses a distributed system?
&gt;&gt;&gt;&gt;
&gt;&gt;&gt;&gt; For the errors below, I checked the directories and the files
are there. Am I not sure what went wrong and how to set the conf to not have central file
system. Thank you.
&gt;&gt;&gt;&gt;
&gt;&gt;&gt;&gt; Error message
&gt;&gt;&gt;&gt; CODE
&gt;&gt;&gt;&gt; w1153435@n51:~/hadoop-0.20.2_cluster&gt;  bin/start-dfs.sh
&gt;&gt;&gt;&gt; bin/start-dfs.sh: line 28: /w1153435/hadoop-0.20.2_cluster/bin/hadoop-config.sh:
No such file or directory
&gt;&gt;&gt;&gt; bin/start-dfs.sh: line 50: /w1153435/hadoop-0.20.2_cluster/bin/hadoop-daemon.sh:
No such file or directory
&gt;&gt;&gt;&gt; bin/start-dfs.sh: line 51: /w1153435/hadoop-0.20.2_cluster/bin/hadoop-daemons.sh:
No such file or directory
&gt;&gt;&gt;&gt; bin/start-dfs.sh: line 52: /w1153435/hadoop-0.20.2_cluster/bin/hadoop-daemons.sh:
No such file or directory
&gt;&gt;&gt;&gt; CODE
&gt;&gt;&gt;
&gt;&gt;&gt; there's  No such file or directory as
&gt;&gt;&gt; /w1153435/hadoop-0.20.2_cluster/bin/hadoop-daemons.sh
&gt;&gt;&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; There is, I checked as shown
&gt;&gt;&gt; w1153435@n51:~/hadoop-0.20.2_cluster&gt;  ls bin
&gt;&gt;&gt; hadoop            rcc                start-dfs.sh      stop-dfs.sh
&gt;&gt;&gt; hadoop-config.sh  slaves.sh          start-mapred.sh  stop-mapred.sh
&gt;&gt;&gt; hadoop-daemon.sh  start-all.sh      stop-all.sh
&gt;&gt;&gt; hadoop-daemons.sh  start-balancer.sh  stop-balancer.sh
&gt;
&gt;try "pwd" to print out where the OS thinks you are, as it doesn't seem 
&gt;to be where you think you are
&gt;
&gt;
&gt;w1153435@ngs:~/hadoop-0.20.2_cluster&gt; pwd
&gt;/home/w1153435/hadoop-0.20.2_cluster
&gt;
&gt;
&gt;w1153435@ngs:~/hadoop-0.20.2_cluster/bin&gt; pwd
&gt;/home/w1153435/hadoop-0.20.2_cluster/bin
&gt;
&gt;&gt;&gt;
&gt;&gt;&gt;
&gt;&gt;&gt;
&gt;&gt;&gt;
&gt;&gt;&gt;&gt;
&gt;&gt;&gt;&gt; I had tried running this command below earlier but also got
problems:
&gt;&gt;&gt;&gt; CODE
&gt;&gt;&gt;&gt; w1153435@ngs:~/hadoop-0.20.2_cluster&gt;  export HADOOP_CONF_DIR=${HADOOP_HOME}/conf
&gt;&gt;&gt;&gt; w1153435@ngs:~/hadoop-0.20.2_cluster&gt;  export HADOOP_SLAVES=${HADOOP_CONF_DIR}/slaves
&gt;&gt;&gt;&gt; w1153435@ngs:~/hadoop-0.20.2_cluster&gt;  ${HADOOP_HOME}/bin/slaves.sh
"mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop"
&gt;&gt;&gt;&gt; -bash: /bin/slaves.sh: No such file or directory
&gt;&gt;&gt;&gt; w1153435@ngs:~/hadoop-0.20.2_cluster&gt;  export HADOOP_HOME=/home/w1153435/hadoop-0.20.2_cluster
&gt;&gt;&gt;&gt; w1153435@ngs:~/hadoop-0.20.2_cluster&gt;  ${HADOOP_HOME}/bin/slaves.sh
"mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop"
&gt;&gt;&gt;&gt; cat: /conf/slaves: No such file or directory
&gt;&gt;&gt;&gt; CODE
&gt;&gt;&gt;&gt;
&gt;&gt;&gt; there's  No such file or directory as /conf/slaves because you set
&gt;&gt;&gt; HADOOP_HOME after setting the other env variables, which are expanded
at
&gt;&gt;&gt; set-time, not run-time.
&gt;&gt;&gt;
&gt;&gt;&gt; I redid the command but still have errors on the slaves
&gt;&gt;&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; w1153435@n51:~/hadoop-0.20.2_cluster&gt;  export HADOOP_HOME=/home/w1153435/hadoop-0.20.2_cluster
&gt;&gt;&gt; w1153435@n51:~/hadoop-0.20.2_cluster&gt;  export HADOOP_CONF_DIR=${HADOOP_HOME}/conf
&gt;&gt;&gt; w1153435@n51:~/hadoop-0.20.2_cluster&gt;  export HADOOP_SLAVES=${HADOOP_CONF_DIR}/slaves
&gt;&gt;&gt; w1153435@n51:~/hadoop-0.20.2_cluster&gt;  ${HADOOP_HOME}/bin/slaves.sh
"mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop"
&gt;&gt;&gt; privn51: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&gt; privn58: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&gt; privn52: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&gt; privn55: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&gt; privn57: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&gt; privn54: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&gt; privn53: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;&gt;&gt; privn56: bash: mkdir -p /home/w1153435/hadoop-0.20.2_cluster/tmp/hadoop:
No such file or directory
&gt;
&gt;try ssh-ing in, do it by hand, make sure you have the right permissions etc
&gt;
&gt;
&gt;I reset the above path variables again and checked that they existed and tried the
command above but same error. I used ssh with no problems and no password request so that
is fine. What else could be wrong?
&gt;w1153435@ngs:~/hadoop-0.20.2_cluster&gt; echo $HADOOP_HOME                   
     /home/w1153435/hadoop-0.20.2_cluster
&gt;w1153435@ngs:~/hadoop-0.20.2_cluster&gt; echo $HADOOP_CONF_DIR               
     /home/w1153435/hadoop-0.20.2_cluster/conf
&gt;w1153435@ngs:~/hadoop-0.20.2_cluster&gt; echo $HADOOP_SLAVES                 
     /home/w1153435/hadoop-0.20.2_cluster/conf/slaves
&gt;w1153435@ngs:~/hadoop-0.20.2_cluster&gt;
&gt;
&gt;
&gt;
&gt;
&gt;


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message