Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B2E8398F9 for ; Mon, 19 Sep 2011 13:10:31 +0000 (UTC) Received: (qmail 90336 invoked by uid 500); 19 Sep 2011 13:10:31 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 90297 invoked by uid 500); 19 Sep 2011 13:10:31 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 90289 invoked by uid 99); 19 Sep 2011 13:10:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Sep 2011 13:10:31 +0000 X-ASF-Spam-Status: No, hits=4.7 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of siddharth.tiwari@live.com designates 65.55.90.149 as permitted sender) Received: from [65.55.90.149] (HELO snt0-omc3-s10.snt0.hotmail.com) (65.55.90.149) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Sep 2011 13:10:23 +0000 Received: from SNT142-W41 ([65.55.90.137]) by snt0-omc3-s10.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 19 Sep 2011 06:10:03 -0700 Message-ID: Content-Type: multipart/alternative; boundary="_6257c5ec-ccd4-4ad9-853a-160e4c9c6be8_" X-Originating-IP: [101.212.89.32] From: Siddharth Tiwari To: hive user list Subject: RE: Decommission of datanode(Urgent) Date: Mon, 19 Sep 2011 13:10:02 +0000 Importance: Normal In-Reply-To: References: MIME-Version: 1.0 X-OriginalArrivalTime: 19 Sep 2011 13:10:03.0085 (UTC) FILETIME=[71763FD0:01CC76CD] --_6257c5ec-ccd4-4ad9-853a-160e4c9c6be8_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi vikas=2C Please include your exclude file in hdfs-site.xml under the property dfs.ho= sts.exclude. please make note that this file must be included at the namenode startup an= d then u hv to edit it for the datanodes you want to exclude. Secondly=2C a= fter editin it for the ip:port of the retiring datanode=2C use the command = bin/hadoop dfsadmin -refreshNodes.=20 also=2C Decommission does not happen momentarily since it requires replicat= ion=20 of potentially a large number of blocks and we do not want the cluster=20 to be overwhelmed with just this one job. The decommission progress can=20 be monitored on the name-node Web UI. Until all blocks are replicated=20 the node will be in "Decommission In Progress" state. When decommission=20 is done the state will change to "Decommissioned". The nodes can be=20 removed whenever decommission is finished. The decommission process can be terminated at any time by editing the confi= guration or the exclude files and repeating the -refreshNodes command. hope it helps. *------------------------* Cheers !!! Siddharth Tiwari TCS world wide Data warehouse and Analytic Team - Americas Have a refreshing day !!! Date: Mon=2C 19 Sep 2011 12:50:13 +0530 Subject: Decommission of datanode(Urgent) From: vikas.srivastava@one97.net To: user@hive.apache.org=3B ayonsinha@yahoo.com=3B viral.bajaria@gmail.com CC: nitin2.kumar@one97.net=3B abhinav.mishra@one97.net Hey folks !! i tried to decommission datanode from hadoop cluster... steps i followed 1: add this in core site dfs.hosts.exclude /home/hadoop/excludes true 1: add this in mapred-site mapred.hosts.exclude /home/hadoop/excludes true 3:create a excludes file and add ip:port in that exp: 10.0.3.31:50010 4: run cmd hadoop dfsadmin -refreshNodes 5: After that my live nodes became 0 and all nodes became dead.. i checked = namenode logs where i found these error msgs 2011-09-19 12:33:47=2C695 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 24 on 9000=2C call sendHeartbeat(DatanodeRegistration(10.0.3.16:50010= =2C storageID=3DDS-1703098060-10.0.3.16-50010-1298269611944=2C infoPort=3D5= 0075=2C ipcPort=3D50020)=2C 2012206694400=2C 1650194042865=2C 271003275264= =2C 0=2C 1) from 10.0.3.16:38587: error: org.apache.hadoop.hdfs.server.prot= ocol.DisallowedDatanodeException: Datanode denied communication with nameno= de: 10.0.3.16:50010 org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanod= e denied communication with namenode: 10.0.3.16:50010 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartb= eat(FSNamesystem.java:2235) at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(Na= meNode.java:704) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod= AccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) 2011-09-19 12:33:47=2C701 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 7 on 9000=2C call sendHeartbeat(DatanodeRegistration(10.0.5.36:50010= =2C storageID=3DDS-809855347-10.0.5.36-50010-1316252293924=2C infoPort=3D50= 075=2C ipcPort=3D50020)=2C 1938687860736=2C 1390486994944=2C 457712619520= =2C 0=2C 1) from 10.0.5.36:58924: error: org.apache.hadoop.hdfs.server.prot= ocol.DisallowedDatanodeException: Datanode denied communication with nameno= de: 10.0.5.36:50010 org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanod= e denied communication with namenode: 10.0.5.36:50010 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartb= eat(FSNamesystem.java:2235) at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(Na= meNode.java:704) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod= AccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) please suggest any help would be appreciated!!!!!!!!!! --=20 With Regards Vikas Srivastava DWH & Analytics TeamMob:+91 9560885900 One97 | Let's get talking ! = --_6257c5ec-ccd4-4ad9-853a-160e4c9c6be8_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi vikas=2C

Please include your exclude file in hdfs-site.xml= under the property dfs.hosts.exclude.
please make note that this= file must be included at the namenode startup and then u hv to edit it for= the datanodes you want to exclude. Secondly=2C after editin it for the ip:= port of the retiring datanode=2C use the command bin/hadoop dfsadmin -re= freshNodes.

also=2C Decommission does not happen momentarily si= nce it requires replication=20 of potentially a large number of blocks and we do not want the cluster=20 to be overwhelmed with just this one job. The decommission progress can=20 be monitored on the name-node Web UI. Until all blocks are replicated=20 the node will be in "Decommission In Progress" state. When decommission=20 is done the state will change to "Decommissioned". The nodes can be=20 removed whenever decommission is finished.

The decommission process can be terminated at any time by editing the confi= guration or the exclude files and repeating the -refreshNodes comman= d.

hope it helps.

*------------= ------------*
Cheers !!!
Siddharth Tiwari
TCS world wide Dat= a warehouse and Analytic Team - Americas
Have a refreshing day !!!

=
Date: Mon=2C 19 Sep 2011 12:50:13 +0530Subject: Decommission of datanode(Urgent)
From: vikas.srivastava@one97.= net
To: user@hive.apache.org=3B ayonsinha@yahoo.com=3B viral.bajaria@gma= il.com
CC: nitin2.kumar@one97.net=3B abhinav.mishra@one97.net

Hey folks !!

i tried to decommission datanode from hadoop c= luster... steps i followed

1: add =3B this in core site
 <=3Bproperty>=3B
    <=3Bname>=3B=
dfs.hosts.exclude<=3B/name>=3B
    <=3Bvalue>=3B/home/hadoop/exc=
ludes<=3B/value>=3B
    <=3Bfinal>=3Btrue<=3B/final>=3B
 =
 <=3B/property>=3B


1: add =3B this in mapred-site
  <=3Bproperty>=3B
 =
   <=3Bname>=3Bmapred.hosts.exclude<=3B/name>=3B

    <=3Bvalue>=3B/home/hadoop/excludes<=3B/value>=3B
    <=3Bf=
inal>=3Btrue<=3B/final>=3B
  <=3B/property>=3B

3:= create a excludes file and add ip:port in that

exp: 10.0.3.31:= 50010

4: run cmd

hadoop dfsadmin -refreshNodes


5: After that my live nodes became 0 and all n= odes became dead.. i checked namenode logs where i found these error msgs

2011-09-19 12:33:47=2C695 INFO org.apache.hadoop.ipc.Server: IPC Server= handler 24 on 9000=2C call sendHeartbeat(DatanodeRegistration(10.0.3.16:50010=2C storageID= =3DDS-1703098060-10.0.3.16-50010-1298269611944=2C infoPort=3D50075=2C ipcPo= rt=3D50020)=2C 2012206694400=2C 1650194042865=2C 271003275264=2C 0=2C 1) fr= om 10.0.3.16:38587= : error: org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException= : Datanode denied communication with namenode: 10.0.3.16:50010
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanod= e denied communication with namenode: 10.0.3.16:50010
 =3B =3B =3B =3B&nbs= p=3B =3B =3B at org.apache.hadoop.hdfs.server.namenode.FSNamesystem= .handleHeartbeat(FSNamesystem.java:2235)
 =3B =3B =3B =3B =3B =3B =3B at org.apache.hado= op.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:704)
 = =3B =3B =3B =3B =3B =3B =3B at sun.reflect.Generate= dMethodAccessor4.invoke(Unknown Source)
 =3B =3B =3B =3B=  =3B =3B =3B at sun.reflect.DelegatingMethodAccessorImpl.invoke= (DelegatingMethodAccessorImpl.java:25)
 =3B =3B =3B =3B =3B =3B =3B at java.lang.refle= ct.Method.invoke(Method.java:597)
 =3B =3B =3B =3B = =3B =3B =3B at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)<= br> =3B =3B =3B =3B =3B =3B =3B at org.apache.h= adoop.ipc.Server$Handler$1.run(Server.java:959)
 =3B =3B =3B=  =3B =3B =3B =3B at org.apache.hadoop.ipc.Server$Handler$1.= run(Server.java:955)
 =3B =3B =3B =3B =3B =3B =3B at java.security.A= ccessController.doPrivileged(Native Method)
 =3B =3B =3B&nbs= p=3B =3B =3B =3B at javax.security.auth.Subject.doAs(Subject.ja= va:396)
 =3B =3B =3B =3B =3B =3B =3B at org.= apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2011-09-19 12:33:4= 7=2C701 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9000=2C = call sendHeartbeat(DatanodeRegistration(10.0.5.36:50010=2C storageID=3DDS-809855347-10.0.5.36= -50010-1316252293924=2C infoPort=3D50075=2C ipcPort=3D50020)=2C 19386878607= 36=2C 1390486994944=2C 457712619520=2C 0=2C 1) from 10.0.5.36:58924: error: org.apache.hadoop= .hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communic= ation with namenode: 1= 0.0.5.36:50010
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanod= e denied communication with namenode: 10.0.5.36:50010
 =3B =3B =3B =3B&nbs= p=3B =3B =3B at org.apache.hadoop.hdfs.server.namenode.FSNamesystem= .handleHeartbeat(FSNamesystem.java:2235)
 =3B =3B =3B =3B =3B =3B =3B at org.apache.hado= op.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:704)
 = =3B =3B =3B =3B =3B =3B =3B at sun.reflect.Generate= dMethodAccessor4.invoke(Unknown Source)
 =3B =3B =3B =3B=  =3B =3B =3B at sun.reflect.DelegatingMethodAccessorImpl.invoke= (DelegatingMethodAccessorImpl.java:25)
 =3B =3B =3B =3B =3B =3B =3B at java.lang.refle= ct.Method.invoke(Method.java:597)
 =3B =3B =3B =3B = =3B =3B =3B at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)<= br> =3B =3B =3B =3B =3B =3B =3B at org.apache.h= adoop.ipc.Server$Handler$1.run(Server.java:959)
 =3B =3B =3B=  =3B =3B =3B =3B at org.apache.hadoop.ipc.Server$Handler$1.= run(Server.java:955)
 =3B =3B =3B =3B =3B =3B =3B at java.security.A= ccessController.doPrivileged(Native Method)
 =3B =3B =3B&nbs= p=3B =3B =3B =3B at javax.security.auth.Subject.doAs(Subject.ja= va:396)
 =3B =3B =3B =3B =3B =3B =3B at org.= apache.hadoop.ipc.Server$Handler.run(Server.java:953)




please suggest any help would = be appreciated!!!!!!!!!!


--
With Regards
Vikas Srivastava

DWH &=3B Analytics Team
Mob= :+91 9560885900
One97 | Let's get talking !

=
= --_6257c5ec-ccd4-4ad9-853a-160e4c9c6be8_--