hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kelvin Rawls <kel...@iswcorp.com>
Subject RE: Configuring Hadoop Cluster
Date Thu, 12 Aug 2010 17:21:30 GMT
Be sure firewall is off OR configured to allow access between master/lsave1/slave2

I use this on Ubuntu 10.04:

#!/bin/bash

# /sbin/iptables -X
# /sbin/iptables -t nat -F
# /sbin/iptables -t nat -X
# /sbin/iptables -t mangle -F
# /sbin/iptables -t mangle -X
/sbin/iptables -P INPUT ACCEPT
/sbin/iptables -P FORWARD ACCEPT
/sbin/iptables -P OUTPUT ACCEPT
/sbin/iptables -F



Kelvin

++++++++++++

Hej,

I am configuring a Hadoop cluster with 1 master
(jobtracker+namenode+secondary namenode) and 2 slaves
(tasktracker+datanode). The 2 datanodes logs do not show any information but
the datanodes processes are running (checked with ps aux).

INITIALIZATION SCRIPT
bin/hadoop namenode -format
bin/start-all

The jobtracker WUI shows always state INITIALIZING with 0 nodes. However, if
I set only the master as a slave the cluster starts and runs successfully.
Can anybody give me any idea of what could be wrong? I attach the following
configuration files:

conf/master

master


conf/slaves

slave1
slave2


conf/core-site.xml

<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://master:9000/</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>

conf/hdfs-site.xml

<configuration>
<property>
  <name>dfs.name.dir</name>
  <value>/nutch/tmp/name/</value>
</property>

<property>
  <name>dfs.data.dir</name>
  <value>/nutch/tmp/data/</value>
</property>

<property>
  <name>dfs.replication</name>
  <value>2</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is
created.
  The default is used if replication is not specified in create time.
  </description>
</property>
</configuration>


conf/mapred-site.xml

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://master:9001</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>

<property>
  <name>mapred.map.tasks</name>
  <value>4</value>
  <description>
    define mapred.map tasks to be number of slave hosts
  </description>
</property>

<property>
  <name>mapred.reduce.tasks</name>
  <value>2</value>
  <description>
    define mapred.reduce tasks to be number of slave hosts
  </description>
</property>

<property>
  <name>mapred.system.dir</name>
  <value>/nutch/tmp/mapreduce/system/</value>
</property>

<property>
  <name>mapred.local.dir</name>
  <value>/nutch/tmp/mapreduce/local/</value>
</property>
</configuration>


Thanks in advance ;)
--
View this message in context: http://lucene.472066.n3.nabble.com/Configuring-Hadoop-Cluster-tp1111545p1111545.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Mime
View raw message