hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: rack topology data update
Date Thu, 13 Sep 2012 03:32:52 GMT
Jameson,

The right process to add a new node with the right mapping is:

1. Update topology file for the new DN.
2. Issue a dfsadmin -refreshNodes to get new topology mapping updated in NN.
3. Start the DN only after (2) so it picks up the right mapping and a
default mapping does not get cached.

On Thu, Sep 13, 2012 at 8:33 AM, Jameson Li <hovlj.ei@gmail.com> wrote:
> Our hadoop version is hadoop-0.20-append+4.
>
> We have configured the rack awareness in the namenode.
> But when I add new datanode, and update the topology data file, and restart
> the datanode, I just see the log in the namenode that:
> 2012-09-13 10:35:25,074 INFO org.apache.hadoop.net.NetworkTopology: Adding a
> new node: /default-rack/ipc:50010
> So should I restart the namenode?
> Is there some command like 'hadoop dfsadmin -refreshtopology'?
>
> My configuration:
>
> core-site.xml:
> <property>
> <name>topology.script.file.name</name>
> <value>conf/rack-awareness.sh</value>
> </property>
> <property>
> <name>topology.script.number.args</name>
> <value>1000</value>
> </property>
>
> conf/rack-awareness.sh:
> #!/bin/sh
>
> HADOOP_CONF=/opt/hadoop/conf
>
> while [ $# -gt 0 ] ; do
>   nodeArg=$1
>   exec< ${HADOOP_CONF}/topology.data
>   result=""
>   while read line ; do
>     ar=( $line )
>     if [ "${ar[0]}" = "$nodeArg" ] ; then
>       result="${ar[1]}"
>     fi
>   done
>   shift
>   if [ -z "$result" ] ; then
>     echo -n "/default-rack "
>   else
>     echo -n "$result "
>   fi
> done
>
> topology.data:
> ipa rackA
> ipb rackA
> ipc rackB
>
>
> And also I have search the mailing list "topology.script.file update":
> I found a mail that:
> Tom Hall 2011-10-27, 16:07
> I was hoping that if I updated the file it would give new answers as
> datanodes were restarted and reconnected but that does not seem to be
> the case.
> Surely I dont need to restart the namenode...
>
> But there is not replying.
> So somebody can help me?
>
>
> 专注于Mysql,MSSQL,Oracle,Hadoop



-- 
Harsh J

Mime
View raw message