Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com
 designates 65.55.34.213 as permitted sender)
Message-ID: <COL117-W29F82B50BDE76780246CD78FB00@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_7ba2b452-731b-4fa5-83b6-1829f04e8c1f_"
From: Michael Segel <michael_segel@hotmail.com>
To: <common-user@hadoop.apache.org>
Subject: RE: decommissioning node woes
Date: Fri, 18 Mar 2011 11:59:19 -0500
Importance: Normal
In-Reply-To: <AANLkTikq9h+ZAAK1EUSOGKPTELs6W6MTxdMZzrv79X6b@mail.gmail.com>
References: 
 <AANLkTikPKGt5zw1QGLse+LPzUDP7Mom=Ty_mXfcuoPhq@mail.gmail.com>,<AANLkTimBeRK_Wf=UYbgs8avVszk1Nzqyc0ZALToGvUp1@mail.gmail.com>,<AANLkTi=QjtzTA_ce-Be_FbfZWhabUPp4ObpTdGR-F2oZ@mail.gmail.com>
 <4E091F9C-FF89-46C2-A288-6340B86D4106@tynt.com>,<AANLkTikq9h+ZAAK1EUSOGKPTELs6W6MTxdMZzrv79X6b@mail.gmail.com>
MIME-Version: 1.0

--_7ba2b452-731b-4fa5-83b6-1829f04e8c1f_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Uhmm...

If you use the default bandwidth allocation and you have a lot of data on t=
he node you want to decommission you can be waiting for weeks before you ca=
n safely take the node out.
If you wanted to=2C you can take the nodes down one by one where you do an =
fsck in between the removal of nodes to get the under replicated blocks ide=
ntified and replicated.
("Normally Namenode
      automatically corrects most of the recoverable failures.")=20

Once you see those blocks successfully replicated... you can take down the =
next.

Is it clean? No=2C not really.
Is it dangerous? No=2C not really.
Do I recommend it? No=2C but its a quick and dirty way of doing things...=20

Or you can up your dfs.balance.bandwidthPerSecIn the configuration files. T=
he default is pretty low.

The downside is that you have to bounce the cloud to get this value updated=
=2C and it could have a negative impact on performance if set too high.

HTH

-Mike


> From: tdunning@maprtech.com
> Date: Fri=2C 18 Mar 2011 09:38:31 -0700
> Subject: Re: decommissioning node woes
> To: common-user@hadoop.apache.org
> CC: james@tynt.com
>=20
> Unless the last copy is on that node.
>=20
> Decommissioning is the only safe way to shut off 10 nodes at once.  Doing
> them one at a time and waiting for replication to (asymptotically) recove=
r
> is painful and error prone.
>=20
> On Fri=2C Mar 18=2C 2011 at 9:08 AM=2C James Seigel <james@tynt.com> wrot=
e:
>=20
> > Just a note.  If you just shut the node off=2C the blocks will replicat=
e
> > faster.
> >
> > James.
> >
> >
> > On 2011-03-18=2C at 10:03 AM=2C Ted Dunning wrote:
> >
> > > If nobody else more qualified is willing to jump in=2C I can at least
> > provide
> > > some pointers.
> > >
> > > What you describe is a bit surprising.  I have zero experience with a=
ny
> > 0.21
> > > version=2C but decommissioning was working well
> > > in much older versions=2C so this would be a surprising regression.
> > >
> > > The observations you have aren't all inconsistent with how
> > decommissioning
> > > should work.  The fact that your nodes look up
> > > after starting the decommissioning isn't so strange.  The idea is tha=
t no
> > > new data will be put on the node=2C nor should it be
> > > counted as a replica=2C but it will help in reading data.
> > >
> > > So that isn't such a big worry.
> > >
> > > The fact that it takes forever and a day=2C however=2C is a big worry=
.  I
> > cannot
> > > provide any help there just off hand.
> > >
> > > What happens when a datanode goes down?  Do you see under-replicated
> > files?
> > > Does the number of such files decrease over time?
> > >
> > > On Fri=2C Mar 18=2C 2011 at 4:23 AM=2C Rita <rmorgan466@gmail.com> wr=
ote:
> > >
> > >> Any help?
> > >>
> > >>
> > >> On Wed=2C Mar 16=2C 2011 at 9:36 PM=2C Rita <rmorgan466@gmail.com> w=
rote:
> > >>
> > >>> Hello=2C
> > >>>
> > >>> I have been struggling with decommissioning data  nodes. I have a 5=
0+
> > >> data
> > >>> node cluster (no MR) with each server holding about 2TB of storage.=
 I
> > >> split
> > >>> the nodes into 2 racks.
> > >>>
> > >>>
> > >>> I edit the 'exclude' file and then do a -refreshNodes. I see the no=
de
> > >>> immediate in 'Decommiosied node' and I also see it as a 'live' node=
!
> > >>> Eventhough I wait 24+ hours its still like this. I am suspecting it=
s a
> > >> bug
> > >>> in my version.  The data node process is still running on the node =
I am
> > >>> trying to decommission. So=2C sometimes I kill -9 the process and I=
 see
> > the
> > >>> 'under replicated' blocks...this can't be the normal procedure.
> > >>>
> > >>> There were even times that I had corrupt blocks because I was impat=
ient
> > >> --
> > >>> waited 24-34 hours
> > >>>
> > >>> I am using 23 August=2C 2010: release 0.21.0 <
> > >>
> > http://hadoop.apache.org/hdfs/releases.html#23+August%2C+2010%3A+releas=
e+0.21.0+available
> > >>>
> > >>> version.
> > >>>
> > >>> Is this a known bug? Is there anything else I need to do to
> > decommission
> > >> a
> > >>> node?
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> --- Get your facts first=2C then you can distort them as you please=
.--
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> --- Get your facts first=2C then you can distort them as you please.=
--
> > >>
> >
> >
 		 	   		  =

--_7ba2b452-731b-4fa5-83b6-1829f04e8c1f_--