Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CF74B17D73 for ; Mon, 6 Apr 2015 06:56:00 +0000 (UTC) Received: (qmail 62267 invoked by uid 500); 6 Apr 2015 06:55:55 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 62146 invoked by uid 500); 6 Apr 2015 06:55:54 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 61964 invoked by uid 99); 6 Apr 2015 06:55:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2015 06:55:54 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [49.212.34.109] (HELO oss.nttdata.co.jp) (49.212.34.109) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2015 06:55:49 +0000 Received: from macaa.local (KD106149150022.au-net.ne.jp [106.149.150.22]) by oss.nttdata.co.jp (Postfix) with ESMTP id 5CB7C17EE2F for ; Mon, 6 Apr 2015 15:55:27 +0900 (JST) Message-ID: <55222DAF.6070603@oss.nttdata.co.jp> Date: Mon, 06 Apr 2015 15:54:39 +0900 From: Akira AJISAKA User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: user@hadoop.apache.org Subject: Re: Hadoop 2.6.0, How to add/remove node to/from running cluster References:

<55201727.3090504@oss.nttdata.co.jp> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.98.6 at oss.nttdata.co.jp X-Virus-Status: Clean X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on oss.nttdata.co.jp X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No, score=-97.6 required=13.0 tests=CONTENT_TYPE_PRESENT, DNS_FROM_AHBL_RHSBL,UNPARSEABLE_RELAY,USER_IN_WHITELIST autolearn=no version=3.2.5 Hi Arthur, > For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML > files should be modified? Just one file. Step1: edit hdfs-site.xml in the NameNode. dfs.hosts.exclude /myhadoop/etc/hadoop/datanode-excludes Step2: edit /myhadoop/etc/hadoop/datanode-excludes in the NameNode, add the full name of the node to be decommissioned Step3: run the following command: $ hdfs dfsadmin -refreshNodes > Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not > use "*hdfs dfsadmin –refreshNodes", * > instead, should use "sbin/*distribute-exclude.sh"* distribute-exclude.sh is to distribute the exclude file to all the NameNodes. If you are running only one NameNode (i.e. you are not using HDFS Federation), you don't need to use this script. > Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or > load main class mradmin" This command is wrong. > Please advise the correct way to decommission a datanode completely (for > both DFS and Nodemanager) in Hadoop 2.x For decommissioning NodeManager, just stopping NodeManager is fine. If you want to decommission NodeManager gracefully, you need to do the following steps. 1. edit yarn-site.xml yarn.resourcemanager.nodes.exclude-path /myhadoop/etc/hadoop/nodemanager-excludes 2. edit /myhadoop/etc/hadoop/nodemanager-excludes in the ResourceManager, add the full name of the node to be decommissioned 3. execute the following command $ yarn rmadmin -refreshNodes Regards, Akira On 4/5/15 09:58, Arthur Chan wrote: > Hi, Thanks. > > > For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML > files should be modified? > > > Could you advise if my following steps are correct? > > > === My Hadoop is 2.6.0 === > Step 1: > edit mapred-site.xml (ADD) >   >  dfs.hosts  > /myhadoop/etc/hadoop/datanode-excludes    >   > >   >  dfs.hosts.exclude  > /myhadoop/etc/hadoop/datanode-excludes    >   > > > step 2: > > edit hdfs-site.xml (ADD) > >   >  mapred.hosts  > /myhadoop/etc/hadoop/datanode-excludes    >   > >   >  mapred.hosts.exclude  > /myhadoop/etc/hadoop/datanode-excludes    >   > > > step 3: > edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all > nodes to it > edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the > node to be decommissioned > > step 4: run the following command > > *hdfs dfsadmin –refreshNodes* > > *hdfs mradmin –refreshNodes * > > > Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not > use "*hdfs dfsadmin –refreshNodes", * > instead, should use "sbin/*distribute-exclude.sh"* > > Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or > load main class mradmin" > > > Please advise the correct way to decommission a datanode completely (for > both DFS and Nodemanager) in Hadoop 2.x > Regards > Arthur > > > On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA > > wrote: > > Hi Arthur, > > > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster? > > Just starting a datanode is fine. > The datanode will be added to the cluster automatically. > > > 2) How to rebalance the cluster after the new node is added? > > Please see > http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Balancer > > > > 3) I could not find the section about how to decommission a datanode > > from a running cluster > > Please see -refreshNodes section of this document. > http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Shell_Commands > > > Regards, > Akira >