Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 51FB79CFE for ; Wed, 30 Nov 2011 04:07:58 +0000 (UTC) Received: (qmail 88273 invoked by uid 500); 30 Nov 2011 04:07:57 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 88209 invoked by uid 500); 30 Nov 2011 04:07:55 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 88201 invoked by uid 99); 30 Nov 2011 04:07:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Nov 2011 04:07:54 +0000 X-ASF-Spam-Status: No, hits=1.6 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kartheek0274@gmail.com designates 209.85.212.48 as permitted sender) Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Nov 2011 04:07:47 +0000 Received: by vbnl22 with SMTP id l22so55947vbn.35 for ; Tue, 29 Nov 2011 20:07:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=XLWLQ8qTCYszj0QhR+reV4bkcN2aUp+ct0bCo42/HEg=; b=viwK4NDsabGivwwAGkM7qMqA38wi92az3NewffKQTitGPXAPXcRpefCRAHZ+vnhy/s KiiP0+6wv0ARKq1BZ93SACkFcmWXQ90L9dlmoQVLOTB2OHOyDplUN8d4rbtJLde2zxCY mncw+WydmI0NRutEQNrLjlYhys7DrQOgX8Ktw= Received: by 10.52.92.210 with SMTP id co18mr536539vdb.111.1322626046426; Tue, 29 Nov 2011 20:07:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.220.156.80 with HTTP; Tue, 29 Nov 2011 20:07:05 -0800 (PST) In-Reply-To: <1542FA4EE20C5048A5C2A3663BED2A6B0F86DC3B@szxeml531-mbx.china.huawei.com> References: <1542FA4EE20C5048A5C2A3663BED2A6B0EEDBBA9@szxeml504-mbs.china.huawei.com> <1542FA4EE20C5048A5C2A3663BED2A6B0F86DC3B@szxeml531-mbx.china.huawei.com> From: kartheek muthyala Date: Wed, 30 Nov 2011 09:37:05 +0530 Message-ID: Subject: Re: Generation Stamp To: hdfs-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf307cfc64aafefe04b2ebdf1d X-Virus-Checked: Checked by ClamAV on apache.org --20cf307cfc64aafefe04b2ebdf1d Content-Type: text/plain; charset=ISO-8859-1 Thanks Uma..:) On Tue, Nov 29, 2011 at 10:48 PM, Uma Maheswara Rao G wrote: > Yes. :-) > ------------------------------ > *From:* kartheek muthyala [kartheek0274@gmail.com] > *Sent:* Tuesday, November 29, 2011 10:20 PM > *To:* hdfs-user@hadoop.apache.org > *Subject:* Re: Generation Stamp > > Uma, first of all thanks for the detailed exemplified explanation. > > So to confirm, the primary use of having this generationTimeStamp is to > ensure consistency of the block?. So, when the pipeline is failed at DN3, > and the client invokes recovery, then the NN will chose DN1 to complete the > pipeline. The DN1 first updates its metafile with the new time stamp, and > then passes this information to the other replica at DN2. Further, in the > future NN sees that this particular block is under replicated and it > assigns some other DNa and asks either DN1/DN2 to replicate the same at > DNa. > > > Thanks, > Kartheek. > > > On Tue, Nov 29, 2011 at 8:10 PM, Uma Maheswara Rao G > wrote: > >> Generationstamp is basically to keep track of the replica states. >> >> Consider one scenario where generation smap will be use: >> >> Create a file which has one block. client started writing that block to >> DN1, DN 2, DN3 ( pipeline ) >> >> After writing some data DN3 failed, then Client will get the exception >> about pipeline failuere. Then Client will handle that exception ( you can >> see it in processDataNodeError in DataStreamer thread) . It will remove DN3 >> and will call the recovery for that block with new generation time stamp, >> then NN will choose one primary DN and assign block synchronization >> work.Then primary DN will ensure that all the remainnng block lengths are >> same ( if require it will truncate to consistant length) and will invoke >> committblckSynchronization. Then remaing datatransfer will resume. >> >> >> >> now block will have new genartion timestamp. You can observe this in >> metadata file for that block in DN. >> >> >> >> now the block will be like blk_12345634444, blk_12345634444_1234.meta >> >> here 1234 is the generation timestamp. >> >> Assume a case, after resuming the write again, DN2 fails, then again >> recovery will starts and will get new Generation time stamp again. now only >> DN1 in pipeline and block is blk_12345634444, blk_12345634444_1235.meta. >> resume the the remaing data writes and complted the last packet. With the >> last packet blocks should be finalized. DN1 is finalized the block >> successfully and DN1 will send blocks received command and block info will >> be updated in blocks map . Assume if DN2 comes back and sending that old >> block in reports to NN. Here NN can find that generation timestamp of that >> block is lesser than DN1 reported blocks genstamp. So, it can take the >> decision now. it can reject the lesser generation time stamp block. >> >> >> >> Yu can see this code in FSNameSystem#addStoredBlock. ofcource there will >> be many conditions like length mismatch..etc >> >> >> >> Hope it will help you.... >> >> >> >> Regards, >> >> Uma >> >> >> >> >> ------------------------------ >> *From:* kartheek muthyala [kartheek0274@gmail.com] >> *Sent:* Tuesday, November 29, 2011 7:44 PM >> *To:* hdfs-user >> *Subject:* Generation Stamp >> >> Hi all, >> Why is there the concept of Generation Stamp that is getting tagged to >> the metadata of the block.? How is it useful? I have seen that in the hdfs >> current directory, the metafiles are tagged with this generation stamp. >> Does this keep track of the versioning? >> ~Kartheek. >> > > --20cf307cfc64aafefe04b2ebdf1d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Uma..:)

On Tue, Nov 29, 2011 at 10= :48 PM, Uma Maheswara Rao G <maheswara@huawei.com> wrote:
Yes. :-)
From: kartheek muthyala [kartheek0274@gmail.com]
Sent: Tuesday, November 29, 2011 10:20 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: Generation Stamp

Uma, first of all thanks for the detailed exemplified explanation.

So to confirm, the primary use of having this generationTimeStamp is to ens= ure consistency of the block?. So, when the pipeline is failed at DN3, and = the client invokes recovery, then the NN will chose DN1 to complete the pip= eline. The DN1 first updates its metafile with the new time stamp, and then passes this information to the = other replica at DN2. Further, in the future NN sees that this particular b= lock is under replicated and it assigns some other DNa and asks either DN1/= DN2 to replicate the same at DNa.


Thanks,
Kartheek.


On Tue, Nov 29, 2011 at 8:10 PM, Uma Maheswara R= ao G <maheswara@hua= wei.com> wrote:

Generationstamp is basically to keep track of the replica states.

=A0Consider one scenario where generation smap will be use:

=A0=A0Create a file which has one block. client started writing that blo= ck to DN1, DN 2, DN3 ( pipeline )

After writing some data DN3 failed, then Client will get the exception a= bout pipeline failuere. Then Client will handle that exception ( you can se= e it in processDataNodeError in DataStreamer thread) . It will remove DN3 a= nd will call the recovery for that block with new generation time stamp, then NN will choose one primary DN a= nd assign block synchronization work.Then primary DN will ensure that all t= he remainnng block lengths are same ( if require it will truncate to consis= tant length) and will invoke committblckSynchronization. Then remaing datatransfer will resume.

=A0

=A0now block will have new genartion timestamp. You can observe this in = metadata file for that block in DN.

=A0

now the block will be like blk_12345634444, blk_12345634444_1234.meta

here 1234 is the generation timestamp.

Assume a case, after resuming the write again, DN2 fails, then again rec= overy will starts and will get new Generation time stamp again. now only DN= 1 in pipeline=A0 and block is blk_12345634444, blk_1= 2345634444_1235.meta. resume the the remaing data writes and complted t= he last packet. With the last packet blocks should be finalized. DN1 is fin= alized the block successfully and DN1 will send blocks received command and block info will be updated in blocks= map . Assume if DN2 comes back and sending that old block in reports to NN= . Here NN can find that generation timestamp of that block is lesser than D= N1 reported blocks genstamp. So, it can take the decision now. it can reject the lesser generation time sta= mp block.

=A0

Yu can see this code in FSNameSystem#addStoredBlock.=A0 ofcource there w= ill be many conditions like length mismatch..etc

=A0

Hope it will help you....

=A0

Regards,

Uma=A0

=A0

=A0

From: kartheek muthyala [kartheek0274@gmail.com]
Sent: Tuesday, November 29, 2011 7:44 PM
To: hdfs-user
Subject: Generation Stamp

Hi all,
Why is there the concept of Generation Stamp that is getting tagged to the = metadata of the block.? How is it useful? I have seen that in the hdfs curr= ent directory, the metafiles are tagged with this generation stamp. Does th= is keep track of the versioning?
~Kartheek.


--20cf307cfc64aafefe04b2ebdf1d--