Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 85966DC76 for ; Tue, 11 Sep 2012 09:12:32 +0000 (UTC) Received: (qmail 87101 invoked by uid 500); 11 Sep 2012 09:12:27 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 87009 invoked by uid 500); 11 Sep 2012 09:12:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 86986 invoked by uid 99); 11 Sep 2012 09:12:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Sep 2012 09:12:26 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Mehul_Choube@symantec.com designates 216.10.195.242 as permitted sender) Received: from [216.10.195.242] (HELO tus1smtoutpex02.symantec.com) (216.10.195.242) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Sep 2012 09:12:19 +0000 X-AuditID: d80ac3f2-b7f066d00000696c-f1-504f005e0395 Received: from ecl1mtahubpin02.ges.symantec.com (ecl1mtahubpin02.ges.symantec.com [10.48.69.202]) by tus1smtoutpex02.symantec.com (Symantec Brightmail Gateway out) with SMTP id 44.BD.26988.E500F405; Tue, 11 Sep 2012 09:11:59 +0000 (GMT) Received: from [155.64.220.137] (helo=TUS1XCHHUBPIN01.SYMC.SYMANTEC.COM) by ecl1mtahubpin02.ges.symantec.com with esmtp (Exim 4.76) (envelope-from ) id 1TBMVi-0007Dk-3w for user@hadoop.apache.org; Tue, 11 Sep 2012 09:11:58 +0000 Received: from APJ1XCHEVSPIN30.SYMC.SYMANTEC.COM ([155.64.220.160]) by TUS1XCHHUBPIN01.SYMC.SYMANTEC.COM ([155.64.220.137]) with mapi; Tue, 11 Sep 2012 02:11:53 -0700 From: Mehul Choube To: "user@hadoop.apache.org" Date: Tue, 11 Sep 2012 02:11:49 -0700 Subject: RE: what happens when a datanode rejoins? Thread-Topic: what happens when a datanode rejoins? Thread-Index: Ac2P/ROILX82N39GTkCNFvPnKPxTCAAAElgQ Message-ID: References: <504EE772.5070909@jp.fujitsu.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_E884EC9CD547324B8976A5D37317AC566D11C7FDD4APJ1XCHEVSPIN_" MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprPIsWRmVeSWpSXmKPExsXCZeB6SjeewT/A4M1sboueKdNYHBg9JnRt YQxgjOKySUnNySxLLdK3S+DKmLv6HHPBl4iK7gNHWBoYP/h1MXJySAiYSNyf0sgGYYtJXLi3 Hsjm4hASeMco0X/3GAuE084kMfnqcmYIZxWjxOGr/awgLWwCehKnlk5h6mLk4BARMJXoeaoL EmYRUJWY9PYSI4gtLGAkse3bIzBbRMBYYuumQywQtpHEvbfHwGxegSiJM1+eQC2bxiTR8eM7 WAOnQKDE9jtfwGxGoPO+n1rDBGIzC4hL3HoynwnibAGJJXvOM0PYohIvH/9jhagXlbjTvp4R oj5f4viLuYwQywQlTs58wjKBUXQWklGzkJTNQlIGEdeRWLD7ExuErS2xbOFrZhj7zIHHTMji CxjZVzHKlJQWGxbnluSXlhSkVhgY6RVX5iYCIy1ZLzk/dxMjMNpucB3+tIPxxlLFQ4wCHIxK PLw7bvoFCLEmlgFVHmKU4GBWEuH1mwUU4k1JrKxKLcqPLyrNSS0+xCjNwaIkzvth91Z/IYH0 xJLU7NTUgtQimCwTB6dUA6PKsVYmhhTxbUr+QiH7950oev/lUOB2y1y1a0+i5k/as83bbcuy iQW8PFkq7rc/SJad4RVVznNdJfXC/1dVU+Sxm9LPQvYtfshtHzc/a+2znYcvNKx2VytpFMq/ G902bXnoB7ZkM3beiPiylIxrcb+MeIxMDkofaHDzn5vCv9cs8IixvMCbOCWW4oxEQy3mouJE ANvPOAiyAgAA X-Virus-Checked: Checked by ClamAV on apache.org --_000_E884EC9CD547324B8976A5D37317AC566D11C7FDD4APJ1XCHEVSPIN_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable >DataNode rejoins take care of only NameNode. Sorry didn't get this From: Narasingu Ramesh [mailto:ramesh.narasingu@gmail.com] Sent: Tuesday, September 11, 2012 2:38 PM To: user@hadoop.apache.org Subject: Re: what happens when a datanode rejoins? Hi Mehul, DataNode rejoins take care of only NameNode. Thanks & Regards, Ramesh.Narasingu On Tue, Sep 11, 2012 at 2:36 PM, Mehul Choube > wrote: > The namenode will asynchronously replicate the blocks to other datanodes = in order to maintain the replication factor after a datanode has not been i= n contact for 10 minutes. What happens when the datanode rejoins after namenode has already re-replic= ated the blocs it was managing? Will namenode ask the datanode to discard the blocks and start managing new= blocks? Or will namenode discard the new blocks which were replicated due to unavai= lability of this datanode? Thanks, Mehul From: George Datskos [mailto:george.datskos@jp.fujitsu.com] Sent: Tuesday, September 11, 2012 12:56 PM To: user@hadoop.apache.org Subject: Re: what happens when a datanode rejoins? Hi Mehul Some of the blocks it was managing are deleted/modified? The namenode will asynchronously replicate the blocks to other datanodes in= order to maintain the replication factor after a datanode has not been in = contact for 10 minutes. The size of the blocks are now modified say from 64MB to 128MB? Block size is a per-file setting so new files will be 128MB, but the old on= es will remain at 64MB. What if the block replication factor was one (yea not in most deployments b= ut say incase) so does the namenode recreate a file once the datanode rejoi= ns? (assuming you didn't perform a decommission) Blocks that lived only on that= datanode will be declared "missing" and the files associated with those bl= ocks will be not be able to be fully read, until the datanode rejoins. George --_000_E884EC9CD547324B8976A5D37317AC566D11C7FDD4APJ1XCHEVSPIN_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

>DataNode rej= oins take care of only NameNode.

Sorry d= idn’t get this

 <= /o:p>

 

From: Narasingu Ramesh [mailto:ramesh.narasingu@gm= ail.com]
Sent: Tuesday, September 11, 2012 2:38 PM
To:= user@hadoop.apache.org
Subject: Re: what happens when a datanode= rejoins?

 =

Hi Mehul,
 =             DataNode= rejoins take care of only NameNode.
Thanks & Regards,
Ramesh.Na= rasingu

On Tue, Sep 11, 2012 at 2:3= 6 PM, Mehul Choube <Mehul_Choube@symantec.com> wrote:

> The namenode will asynchronously replicate= the blocks to other datanodes in order to maintain the replication factor = after a datanode has not been in contact for 10 minutes.

What happens when the datanode rejoins after namenode has already = re-replicated the blocs it was managing?

Will namenode= ask the datanode to discard the blocks and start managing new blocks?=

Or will namenode discard the new blocks which were replica= ted due to unavailability of this datanode?

 

 

 

T= hanks,

Mehul

 

 

<= p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:= auto'>From: George Datskos [mailto:george.datskos@jp.fujitsu.com]
Sent: Tuesday, September 11, 2012 12:56 PM
To: user@hadoop.apache.org
Subj= ect: Re: what happens when a datanode rejoins?

 

Hi Mehul

Some of the blocks it was managing are deleted/modified?

<= /blockquote>


The namenode will asynchronously replicate the blocks to o= ther datanodes in order to maintain the replication factor after a datanode= has not been in contact for 10 minutes.

The s= ize of the blocks are now modified say from 64MB to 128MB?

<= br>Block size is a per-file setting so new files will be 128MB, but the old= ones will remain at 64MB.

What if the block r= eplication factor was one (yea not in most deployments but say incase) so d= oes the namenode recreate a file once the datanode rejoins?

<= p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:= auto'>
(assuming you didn't perform a decommission) Blocks that lived on= ly on that datanode will be declared "missing" and the files asso= ciated with those blocks will be not be able to be fully read, until the da= tanode rejoins.



George

 

= --_000_E884EC9CD547324B8976A5D37317AC566D11C7FDD4APJ1XCHEVSPIN_--