Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3FE98FBF5 for ; Wed, 3 Apr 2013 01:53:48 +0000 (UTC) Received: (qmail 56876 invoked by uid 500); 3 Apr 2013 01:53:43 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 56574 invoked by uid 500); 3 Apr 2013 01:53:43 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 56567 invoked by uid 99); 3 Apr 2013 01:53:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Apr 2013 01:53:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of azuryyyu@gmail.com designates 209.85.223.169 as permitted sender) Received: from [209.85.223.169] (HELO mail-ie0-f169.google.com) (209.85.223.169) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Apr 2013 01:53:39 +0000 Received: by mail-ie0-f169.google.com with SMTP id qd14so1166718ieb.28 for ; Tue, 02 Apr 2013 18:53:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=4GGx5V7Hm2xSqNZoXxYwMSae1Fjif7b6tYeTp61Ire0=; b=s0lkhaoZ/6/urqUzunBu7Ak3FDW7/nHpoqOD71BANfnqOId0YsXfNclHiDxm6YYvw6 VgakRiU0P7TriGK1HRr0RIFeHsAASSeFta+jnWQcxYRoGXFF3TauOMQeYi9cXWZF0Sa2 nDF1FhwEbFvzny5PmRWP0WE4AMDbf2fHZjKgWOOnfitA2iFevXKrBXXYHv0C+OClu61h lfUNE5GgNkJJQC880ZU2kAUYvP10+Z388jTCOqfuJ45RM++bLBnu0TAY+wXxu/n6duKv bHOF0DdLd0AIxpRF72TBrjb98Tm3LlvS1v4PiYx92YnyBtJxpC0Y1oKobCxmzsIURZNB v3FA== MIME-Version: 1.0 X-Received: by 10.50.50.71 with SMTP id a7mr6082253igo.14.1364953998671; Tue, 02 Apr 2013 18:53:18 -0700 (PDT) Received: by 10.64.26.70 with HTTP; Tue, 2 Apr 2013 18:53:18 -0700 (PDT) In-Reply-To: References: <57310067-9C98-41EA-A674-82940D584EED@gmail.com> <9E45CED1-1D86-4116-8892-7C514EF64342@gmail.com> <8E7DDACE-7658-4B34-AAD7-89A5A6D52EF4@gmail.com> Date: Wed, 3 Apr 2013 09:53:18 +0800 Message-ID: Subject: Re: are we able to decommission multi nodes at one time? From: Azuryy Yu To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e013c5c4439c7d604d96b1ef8 X-Virus-Checked: Checked by ClamAV on apache.org --089e013c5c4439c7d604d96b1ef8 Content-Type: text/plain; charset=EUC-KR Content-Transfer-Encoding: quoted-printable bq. then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2. sorry for typo. Correct for it: then namenode start to copy block replicates on DN-1 to another DN, supposed DN-2. On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu wrote: > It's different. > If you just want to stop DN-1 a short time, just kill the DataNode proces= s > on DN-1. then do what you want. during this time, Namenode cannot receiv= e > the heart beat from DN-1, then namenode start to copy block replicates on > DN-2 to another DN, supposed DN-2. > > But when you start DN-1 again, Namenode receive the DN-1 registration, > then namenode stop to copy the DN-1's block replicates even if NN doesn't > finish coping. > > Am I explain clearly? > > > > On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim wrote: > >> @Harsh >> >> What's the reasons to make big gaps for removing nodes between >> decommission and just down nodes? >> In my understanding, both are necessary to copy un-replicated blocks to >> another alive nodes. >> If main costs of them are this one, total elapsed time couldn't be big >> different. >> >> Could you share some articles or documents to understand about >> decommissioning procedures? >> - explaining is always thanks ;) >> >> >> 2013. 4. 2., =BF=C0=C8=C4 5:37, Harsh J =C0=DB=BC= =BA: >> >> > Yes, you can do the downtime work in steps of 2 DNs at a time, >> > especially since you mentioned the total work would be only ~30mins at >> > most. >> > >> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim >> > wrote: >> >> the rest of nodes to be alive has enough size to store. >> >> >> >> for this one that you've mentioned. >> >>> its easier to do so in a rolling manner without need of a >> >>> decommission. >> >> >> >> to check my understanding, just shutting down 2 of them and then 2 >> more and then 2 more without decommissions. >> >> >> >> is this correct? >> >> >> >> >> >> 2013. 4. 2., =BF=C0=C8=C4 4:54, Harsh J =C0=DB= =BC=BA: >> >> >> >>> Note though that its only possible to decommission 7 nodes at the sa= me >> >>> time and expect it to finish iff the remaining 8 nodes have adequate >> >>> free space for the excess replicas. >> >>> >> >>> If you're just going to take them down for a short while (few mins >> >>> each), its easier to do so in a rolling manner without need of a >> >>> decommission. You can take upto two down at a time on a replication >> >>> average of 3 or 3+, and put it back in later without too much data >> >>> movement impact. >> >>> >> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang >> wrote: >> >>>> It's reasonable to decommission 7 nodes at the same time. >> >>>> But may be it also takes long time to finish it. >> >>>> Because all the replicas in these 7 nodes need to be copied to >> remaining 8 >> >>>> nodes. >> >>>> The size of transfer from these nodes to the remaining nodes is >> equal. >> >>>> >> >>>> >> >>>> 2013/4/2 Henry Junyoung Kim >> >>>>> >> >>>>> :) >> >>>>> >> >>>>> currently, I have 15 data nodes. >> >>>>> for some tests, I am trying to decommission until 8 nodes. >> >>>>> >> >>>>> Now, the total dfs used size is 52 TB which is including all >> replicated >> >>>>> blocks. >> >>>>> from 15 to 8, total spent time is almost 4 days long. ;( >> >>>>> >> >>>>> someone mentioned that I don't need to decommission node by node. >> >>>>> for this case, is there no problems if I decommissioned 7 nodes at >> the >> >>>>> same time? >> >>>>> >> >>>>> >> >>>>> 2013. 4. 2., =BF=C0=C8=C4 12:14, Azuryy Yu = =C0=DB=BC=BA: >> >>>>> >> >>>>> I can translate it to native English: how many nodes you want to >> >>>>> decommission? >> >>>>> >> >>>>> >> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang >> wrote: >> >>>>>> >> >>>>>> You want to decommission how many nodes? >> >>>>>> >> >>>>>> >> >>>>>> 2013/4/2 Henry JunYoung KIM >> >>>>>>> >> >>>>>>> 15 for datanodes and 3 for replication factor. >> >>>>>>> >> >>>>>>> 2013. 4. 1., =BF=C0=C8=C4 3:23, varun kumar =C0=DB=BC=BA: >> >>>>>>> >> >>>>>>>> How many nodes do you have and replication factor for it. >> >>>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Harsh J >> >> >> > >> > >> > >> > -- >> > Harsh J >> >> > --089e013c5c4439c7d604d96b1ef8 Content-Type: text/html; charset=EUC-KR Content-Transfer-Encoding: quoted-printable
bq. then namenode start to copy block replicates= on DN-2 to another DN, supposed DN-2.

sorry for typo.
Correct for it:
then namenode start to copy block replicates on D= N-1 to another DN, supposed DN-2.


On Wed, Apr 3= , 2013 at 9:51 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
It's different.
If you just want to= stop DN-1 a short time, just kill the DataNode process on DN-1. then do wh= at you want. during this time, Namenode  cannot receive the heart beat= from DN-1, then namenode start to copy block replicates on DN-2 to another= DN, supposed DN-2.

But when you start DN-1 again, Namenode receive the DN-1 reg= istration, then namenode stop to copy the DN-1's block replicates even = if NN doesn't finish coping.

Am I explain clearly?


On Wed, Apr 3, 2013 at 9:43 AM, = Henry Junyoung Kim <henry.jykim@gmail.com> wrote:
@Harsh

What's the reasons to make big gaps for removing nodes between decommis= sion and just down nodes?
In my understanding, both are necessary to copy un-replicated blocks to ano= ther alive nodes.
If main costs of  them are this one, total elapsed time couldn't b= e big different.

Could you share some articles or documents to understand about decommission= ing procedures?
- explaining is always thanks ;)


2013. 4. 2., =BF=C0=C8=C4 5:37, Harsh J <harsh@cloudera.com> =C0=DB=BC=BA:

> Yes, you can do the downtime work in steps of 2 DNs at= a time,
> especially since you mentioned the total work would be only ~30mins at=
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <henry.j= ykim@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2= more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., =BF=C0=C8=C4 4:54, Harsh J <harsh@cloudera.com> =C0=DB=BC=BA:<= br> >>
>>> Note though that its only possible to decommission 7 nodes at = the same
>>> time and expect it to finish iff the remaining 8 nodes have ad= equate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (= few mins
>>> each), its easier to do so in a rolling manner without need of= a
>>> decommission. You can take upto two down at a time on a replic= ation
>>> average of 3 or 3+, and put it back in later without too much = data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <yanbohappy@gmail.com> wr= ote:
>>>> It's reasonable to decommission 7 nodes at the same ti= me.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copie= d to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nod= es is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <henry.jykim@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I  have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 no= des.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is includi= ng all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. = ;(
>>>>>
>>>>> someone mentioned that I don't need to decommissio= n node by node.
>>>>> for this case, is there no problems if I decommissione= d 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., =BF=C0=C8=C4 12:14, Azuryy Yu <azuryyyu@gmail.com>= =C0=DB=BC=BA:
>>>>>
>>>>> I can translate it to native English: how many nodes y= ou want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <yanbohappy@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <henry.jykim@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.=
>>>>>>>
>>>>>>> 2013. 4. 1., =BF=C0=C8=C4 3:23, varun kumar &l= t;varun.uid@gmail.= com> =C0=DB=BC=BA:
>>>>>>>
>>>>>>>> How many nodes do you have and replication= factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J



--089e013c5c4439c7d604d96b1ef8--