hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arthur Chan <arthur.hk.c...@gmail.com>
Subject Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Date Sun, 05 Apr 2015 00:58:44 GMT
Hi,  Thanks.


For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
files should be modified?


Could you advise if my following steps are correct?


=== My Hadoop is 2.6.0 ===
Step 1:
edit mapred-site.xml (ADD)
<property>
  <name>dfs.hosts</name>
  <value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>

<property>
  <name>dfs.hosts.exclude</name>
  <value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>


step 2:

edit hdfs-site.xml (ADD)
<property>
  <name>mapred.hosts</name>
  <value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>

<property>
  <name>mapred.hosts.exclude</name>
  <value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>


step 3:
edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
nodes to it
edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the node
to be decommissioned

step 4:  run the following command

*hdfs dfsadmin –refreshNodes*

*hdfs mradmin –refreshNodes  *

Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
use "*hdfs
dfsadmin –refreshNodes", *
instead, should use "sbin/*distribute-exclude.sh"*

Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
load main class mradmin"


Please advise the correct way to decommission a datanode completely (for
both DFS and Nodemanager)  in Hadoop 2.x
Regards
Arthur


On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA <ajisakaa@oss.nttdata.co.jp>
wrote:

> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Balancer
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
>
> Regards,
> Akira
>
>

Mime
View raw message