Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1CABEE2B9 for ; Mon, 24 Dec 2012 10:51:30 +0000 (UTC) Received: (qmail 65288 invoked by uid 500); 24 Dec 2012 10:51:25 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 65147 invoked by uid 500); 24 Dec 2012 10:51:25 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 65132 invoked by uid 99); 24 Dec 2012 10:51:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Dec 2012 10:51:24 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ablozhou@gmail.com designates 209.85.215.43 as permitted sender) Received: from [209.85.215.43] (HELO mail-la0-f43.google.com) (209.85.215.43) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Dec 2012 10:51:19 +0000 Received: by mail-la0-f43.google.com with SMTP id z14so8262101lag.2 for ; Mon, 24 Dec 2012 02:50:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=daOlZZxnwmqeHuN9tXBPN3hOw2yHP6L5QXRVTDw7ig8=; b=jn8kz4K7lNNQt8jKOCRBJ/yJ36CKgXu26TMn1ZQJ2K+uYvIxVWwOXiusO+bWX99nhB 7bU8QZbZK2JttSHc4s+5TqhpzIZPepyrT9RYeAxQi9LSKjZvmssfLLbXKwQAlfPjYshZ tVCfpnaKBU+6+qKsFUM2eJ74ACwqX0r68M1rZQSkaNrFjxDzHKk5hJ+PAVBHHzPD/xJN C6ZkcJPajx0LG/mBY1rhXJoWVNZqf31GkdQhgWhstkDz6incVE/sh2GP1bHsNmWzIxRZ gkheM2igCf+8whxEj39HKm8JoOcl5DtRTiRO9Pt1YYQn3iUSA9z7xSKM1z3loId7J/nL yV3w== Received: by 10.152.125.237 with SMTP id mt13mr19889972lab.45.1356346257063; Mon, 24 Dec 2012 02:50:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.87.231 with HTTP; Mon, 24 Dec 2012 02:50:35 -0800 (PST) In-Reply-To: References: From: =?UTF-8?B?5ZGo5qKm5oOz?= Date: Mon, 24 Dec 2012 18:50:35 +0800 Message-ID: Subject: Re: why not hadoop backup name node data to local disk daily or hourly? To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d04426cccd83e8504d196f8f2 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04426cccd83e8504d196f8f2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Tariq=EF=BC=8C Thanks for your patient. I know that fsimage stores metadata of blocks. I have three machine to back it, so I don't worry about it lost. I'm using SNN and NFS to backup NN data file. But as the description above, my damaged data dirtied every nodes that I backed up automatically. BTW: you looks like the actor of PI on the movie "lifes of PI":) Best regards, Andy Zhou 2012/12/20 Mohammad Tariq > Hello Andy, > > NN stores all the metadata in a file called as "fsimage". The > fsimage file contains a snapshot of the HDFS metadata. Along with fsimage > NN also holds "edit log" files. Whenever there is a change to HDFS, it > gets appended to the edits file. When these log files grow big, they are > merged together with fsimage file. These files are stored on the local FS > at the path specified by the "dfs.name.dir" property in "hdfs-site.xml" > file. To prevent any loss you can give multiple locations as the value fo= r > this property, say 1 on your local disk and another on a network drive in > case you HD get crashed you still have the metadata safe with you in that > network drive.(The condition which you have faced recently) > > Now, coming to the SNN. It is a helper node for the NN. SNN periodically > pulls the fsimage file, which would have grown quite big by now. And the = NN > starts the cycle again. Suppose, you are ruuning completely out of luck a= nd > loose the entire NN. In such a case you can take his copy of fsimage from > the SNN and retrieve your metadata back. > > HTH > > Best Regards, > Tariq > +91-9741563634 > https://mtariq.jux.com/ > > > On Thu, Dec 20, 2012 at 3:18 PM, =E5=91=A8=E6=A2=A6=E6=83=B3 wrote: > >> Some reasons lead to my name node data error, but the error data also >> overwrite the second name node data, also the NFS backup. I want to reco= ver >> the name node data a day ago or even a week ago,but I can't. I have to b= ack >> up name node data manually or write a bash script to backup it? why had= oop >> does not give a configure to backup name node data to local disk daily= or >> hourly with different time stamp name? >> >> The same question is to HBase's .META. and -ROOT- table. I think it's >> history storage is more important 100 times than the log history. >> >> I think it could be implemented in Second Name Node/Check Points Node or >> Back Node. Now I do this just using bash script. >> >> Some one agree with me? >> >> >> Best Regards, >> Andy Zhou >> > > --f46d04426cccd83e8504d196f8f2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi=E3=80=80Tariq=EF=BC=8C
Thanks for your patient. I know that fsimage = stores metadata of blocks. I have three machine to back it, so I don't = worry about it lost. I'm using SNN and NFS to backup NN data file. But = as the description above, my damaged data dirtied every nodes that I backed= up automatically.

BTW: you looks like the actor of PI on the movie "= lifes of PI":)
Best regards,
Andy Zhou

2012/12/20 Mohammad Tariq <dontariq@gmail.com&= gt;
Hello Andy,

<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 NN stores all the metadata in= a file called as "fsimage".=C2=A0The fsimage file contains a sna= pshot of the HDFS metadata. Along with fsimage NN also holds =C2=A0"ed= it log" files.=C2=A0Whenever there is a change to HDFS, it gets append= ed to the edits file. When these log files grow big, they are merged togeth= er with fsimage file. These files are stored on the local FS at the path sp= ecified by the "dfs.name.dir" property in "hdfs-site.xml&quo= t; file. To prevent any loss you can give multiple locations as the value f= or this property, say 1 on your local disk and another on a network drive i= n case you HD get crashed you still have the metadata safe with you in that= network drive.(The condition which you have faced recently)

Now, coming to the SNN. It is a helper node for the NN.= SNN periodically pulls the fsimage file, which would have grown quite big = by now. And the NN starts the cycle again. Suppose, you are ruuning complet= ely out of luck and loose the entire NN. In such a case you can take his co= py of fsimage from the SNN and retrieve your metadata back.

HTH



On Thu, Dec 20, 2012 a= t 3:18 PM, =E5=91=A8=E6=A2=A6=E6=83=B3 <ablozhou@gmail.com>= wrote:
Some reasons lead to my name node data error, but the error data also overw= rite the second name node data, also the NFS backup. I want to recover the = name node data a day ago or even a week ago,but I can't. I have to back= up name node data manually or write a bash script to backup it?=C2=A0why = =C2=A0hadoop does not give a configure to =C2=A0 backup name node data to l= ocal disk daily or =C2=A0hourly with different time stamp name?=C2=A0

The same question is to HBase's .META. and -ROOT- table.= I think it's history storage is more important 100 =C2=A0times than th= e log history.

I think it could be implemented in = Second Name Node/Check Points Node or Back Node. Now I do this just using b= ash script.

Some one agree with me?=C2=A0

=
Best Regards,
Andy Zhou


--f46d04426cccd83e8504d196f8f2--