hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akira AJISAKA <ajisa...@oss.nttdata.co.jp>
Subject Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Date Mon, 06 Apr 2015 06:54:39 GMT
Hi Arthur,

 > For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
 > files should be modified?

Just one file.

Step1: edit hdfs-site.xml in the NameNode.

<property>
   <name>dfs.hosts.exclude</name>
   <value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>

Step2: edit /myhadoop/etc/hadoop/datanode-excludes in the NameNode,
add the full name of the node to be decommissioned

Step3: run the following command:

$ hdfs dfsadmin -refreshNodes

 > Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
 > use "*hdfs dfsadmin –refreshNodes", *
 > instead, should use "sbin/*distribute-exclude.sh"*

distribute-exclude.sh is to distribute the exclude file
to all the NameNodes. If you are running only one NameNode
(i.e. you are not using HDFS Federation), you don't need to
use this script.

 > Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
 > load main class mradmin"

This command is wrong.

 > Please advise the correct way to decommission a datanode completely (for
 > both DFS and Nodemanager)  in Hadoop 2.x

For decommissioning NodeManager, just stopping NodeManager is fine.
If you want to decommission NodeManager gracefully, you need to do
the following steps.

1. edit yarn-site.xml

<property>
   <name>yarn.resourcemanager.nodes.exclude-path</name>
   <value>/myhadoop/etc/hadoop/nodemanager-excludes</value>
</property>

2. edit /myhadoop/etc/hadoop/nodemanager-excludes
in the ResourceManager, add the full name of the node
to be decommissioned

3. execute the following command

$ yarn rmadmin -refreshNodes

Regards,
Akira

On 4/5/15 09:58, Arthur Chan wrote:
> Hi,  Thanks.
>
>
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
>
>
> Could you advise if my following steps are correct?
>
>
> === My Hadoop is 2.6.0 ===
> Step 1:
> edit mapred-site.xml (ADD)
> <property>

>    
<name>dfs.hosts</name>

>    <value>/myhadoop/etc/hadoop/datanode-excludes</value>
 

> </property>

>
> <property>

>    
<name>dfs.hosts.exclude</name>

>    <value>/myhadoop/etc/hadoop/datanode-excludes</value>
 

> </property>

>
>
> step 2:
>
> edit hdfs-site.xml (ADD)
>
> <property>

>    
<name>mapred.hosts</name>

>    <value>/myhadoop/etc/hadoop/datanode-excludes</value>
 

> </property>

>
> <property>

>    
<name>mapred.hosts.exclude</name>

>    <value>/myhadoop/etc/hadoop/datanode-excludes</value>
 

> </property>

>
>
> step 3:
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
> nodes to it
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the
> node to be decommissioned
>
> step 4:  run the following command
>
> *hdfs dfsadmin –refreshNodes*
>
> *hdfs mradmin –refreshNodes *
>
>
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
>
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
>
>
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager)  in Hadoop 2.x
> Regards
> Arthur
>
>
> On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA
> <ajisakaa@oss.nttdata.co.jp <mailto:ajisakaa@oss.nttdata.co.jp>> wrote:
>
>     Hi Arthur,
>
>     > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
>     Just starting a datanode is fine.
>     The datanode will be added to the cluster automatically.
>
>     > 2) How to rebalance the cluster after the new node is added?
>
>     Please see
>     http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Balancer
>     <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer>
>
>     > 3) I could not find the section about how to decommission a datanode
>     > from a running cluster
>
>     Please see -refreshNodes section of this document.
>     http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Shell_Commands
>     <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands>
>
>     Regards,
>     Akira
>


Mime
View raw message