Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of gerritjvv@googlemail.com
 designates 209.85.161.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QZaqfVy/BC/Wt0D0sMwXI6Vq/Oo0DW2aXXjNvkWLDv3gwJVKxdGAk4ZZh/QSwZwS4a
         lIK7ftpO9oGugxler9h6ZIYq2fI+/InKJQq3jxEdKmZD/SpbwXCNeZ7LRcAq6uSysuxu
         kre8ae1FrhASe08HHVKRnhNVlQ1+Pb8tMMGDo=
MIME-Version: 1.0
In-Reply-To: <EBE923B5F1EB42A4BE1BE74BD4A776A7@xPC>
References: <EBE923B5F1EB42A4BE1BE74BD4A776A7@xPC>
Date: Thu, 18 Nov 2010 10:30:29 +0000
Message-ID: <AANLkTi=mY6CtLUZKvbKha_OL5YXyTnLpb-Gng1KyLa_9@mail.gmail.com>
Subject: Re: Namenode Role
From: Gerrit Jansen van Vuuren <gerritjvv@googlemail.com>
To: hdfs-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba47680d6a55d80495514752

--90e6ba47680d6a55d80495514752
Content-Type: text/plain; charset=ISO-8859-1

Hi,

There is some development going on in both yahoo and facebook about making
the namenode HA, but so far there is nothing released that will do this.
So to answer your question: no, the namenode is a single point of failure
with no possibility of switching during runtime.

The only solution is to:
-> write output namenode metadata to two locations: localdisk, and a ntfs
mount.
-> you must always run the seconday/checkpoint namenode. if not the Namenode
will never merge its edits.log file into the on disk image file. (the
primary namenode only merges edits.log into the image under two conditions :
restart, or the secondary namenode requests a checkpoint)
-> make backups with a cronned script requesting checkpoints via the
namenode http api, and store these backups off rack even off site.

Using another namenode when the current namenode fails is then a restore
from one of the backups you've made or using one of the checkpoints made by
the secondary namenode. But I can't stress enough the fact that you need to
make as many backups as possible of your metadata, or else total data loss
will occur if you can't recover the metadata.

Hope this helps.

cheers,
 Gerrit

On Thu, Nov 18, 2010 at 3:20 AM, Ozcan ILIKHAN <ilikhan@cs.wisc.edu> wrote:

> Currently in my mini cluster I have one active and one backup NameNode.
> Whenever I need backup NameNode to be active/regular NameNode, I shutdown it
> and restart in active mode. As far as I understand from documentation and
> code, there is no way to switch from backup to active role at run time.
>
> Does anyone have a better idea of handling this situation?
>
> Thanks,
> Ozcan.
>

--90e6ba47680d6a55d80495514752
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,<br><br>There is some development going on in both yahoo and facebook ab=
out making the namenode HA, but so far there is nothing released that will =
do this.<br>So to answer your question: no, the namenode is a single point =
of failure with no possibility of switching during runtime.<br>
<br>The only solution is to:<br>-&gt; write output namenode metadata to two=
 locations: localdisk, and a ntfs mount.<br>-&gt; you must always run the s=
econday/checkpoint namenode. if not the Namenode will never merge its edits=
.log file into the on disk image file. (the primary namenode only merges ed=
its.log into the image under two conditions : restart, or the secondary nam=
enode requests a checkpoint)<br>
-&gt; make backups with a cronned script requesting checkpoints via the nam=
enode http api, and store these backups off rack even off site.<br><br>Usin=
g another namenode when the current namenode fails is then a restore from o=
ne of the backups you&#39;ve made or using one of the checkpoints made by t=
he secondary namenode. But I can&#39;t stress enough the fact that you need=
 to make as many backups as possible of your metadata, or else total data l=
oss will occur if you can&#39;t recover the metadata.<br>
<br>Hope this helps.<br><br>cheers,<br>=A0Gerrit<br><br><div class=3D"gmail=
_quote">On Thu, Nov 18, 2010 at 3:20 AM, Ozcan ILIKHAN <span dir=3D"ltr">&l=
t;<a href=3D"mailto:ilikhan@cs.wisc.edu">ilikhan@cs.wisc.edu</a>&gt;</span>=
 wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; borde=
r-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Currently in my m=
ini cluster I have one active and one backup NameNode. Whenever I need back=
up NameNode to be active/regular NameNode, I shutdown it and restart in act=
ive mode. As far as I understand from documentation and code, there is no w=
ay to switch from backup to active role at run time.<br>

<br>
Does anyone have a better idea of handling this situation?<br>
<br>
Thanks,<br><font color=3D"#888888">
Ozcan. <br>
</font></blockquote></div><br>

--90e6ba47680d6a55d80495514752--