hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jorn Argelo - Ephorus" <Jorn.Arg...@ephorus.com>
Subject RE: Running more than one secondary namenode
Date Wed, 12 Oct 2011 10:29:27 GMT
Hi Chris,

I am doing exactly what you described there apart from the regular
backup thing (which is still on the todo list). Unfortunately my Java
knowledge is poor at best so I'm not sure if I 
would actually understand the Namenode internals. I'm going to give it a
try nevertheless though!

I guess you're quite right that if we have regular backups of the
namenode fsimage and edit logs we're quite safe.

Thanks for your feedback.


-----Oorspronkelijk bericht-----
Van: Chris Smith [mailto:csmithx@gmail.com] 
Verzonden: woensdag 12 oktober 2011 12:03
Aan: common-user@hadoop.apache.org
Onderwerp: Re: Running more than one secondary namenode


If you've configured the Name Node fsimage and edit log replication to
both  NFS and Secondary Name Node and regularly backup the fsimage and
edit logs you would do better investing time in understanding exactly
how the Name Node builds up it's internal database and how it applies
it's edit logs; 'read the code, Luke'.

Then, if you really want to be prepared, you can then produce some
test scenarios by applying a corruption (that the Name Node can't
handle automatically) to the fsimage or edit logs on a sacrificial
system (VM?) and see if you can recover from this.  That way, if you
ever get hit with a Name Node corruption you'll be in a much better
place to recovery most/all of your data.

Even with the best setup it can happen if you hit  a 'corner case'


On 12 October 2011 08:50, Jorn Argelo - Ephorus
<Jorn.Argelo@ephorus.com> wrote:
> Hi all,
> I was wondering if there are any (technical) issues with running two
> secondary namenodes on two separate servers rather than running just
> one. Since basically everything falls or stands with a consistent
> snapshot of the namenode fsimage I was considering to run two
> namenodes for additional resilience. Has this been done before or am I
> being too paranoid? Are there any caveats with doing this?
> Thanks,
> Jorn

View raw message