hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Noguchi <knogu...@yahoo-inc.com>
Subject Re: HDFS Backup nodes
Date Wed, 07 Dec 2011 16:58:51 GMT
Hi Jorn, 

Which hadoop version were you using when you hit that issue?

Koji


On 12/7/11 5:25 AM, "Jorn Argelo - Ephorus" <Jorn.Argelo@ephorus.com> wrote:

> Just to add to that note - we've ran into an issue where the NFS share
> was out of sync (the namenode storage failed even though the NFS share
> was working), but the other local metadata was fine. At the restart of
> the namenode it picked the NFS share's fsimage even if it was out of
> sync. This had the effect that loads of blocks were marked as invalid
> and deleted by the datanodes, and the namenode never came out of safe
> mode because it was missing blocks. The Hadoop documentation says it
> always picks the most recent version of the fsimage but in my case this
> doesn't seem to have happened. Maybe a bug? With that said I've been
> having issues with NFS before (the NFS namenode storage always failed
> every hour even if the cluster was idle).
> 
> Now since this was just test data it wasn't all that important ... but
> if that would happen with your production cluster you got yourself a
> problem. I've moved away from NFS and I'm using DRBD instead. Not having
> any problems anymore whatsoever.
> 
> YMMV.
> 
> Jorn
> 
> -----Oorspronkelijk bericht-----
> Van: Joey Echeverria [mailto:joey@cloudera.com]
> Verzonden: woensdag 7 december 2011 12:08
> Aan: common-user@hadoop.apache.org
> Onderwerp: Re: HDFS Backup nodes
> 
> You should also configure the Namenode to use an NFS mount for one of
> it's storage directories. That will give the most up-to-date back of
> the metadata in case of total node failure.
> 
> -Joey
> 
> On Wed, Dec 7, 2011 at 3:17 AM, praveenesh kumar <praveenesh@gmail.com>
> wrote:
>> This means still we are relying on Secondary NameNode idealogy for
>> Namenode's backup.
>> Can OS-mirroring of Namenode is a good alternative keep it alive all
> the
>> time ?
>> 
>> Thanks,
>> Praveenesh
>> 
>> On Wed, Dec 7, 2011 at 1:35 PM, Uma Maheswara Rao G
> <maheswara@huawei.com>wrote:
>> 
>>> AFAIK backup node introduced in 0.21 version onwards.
>>> ________________________________________
>>> From: praveenesh kumar [praveenesh@gmail.com]
>>> Sent: Wednesday, December 07, 2011 12:40 PM
>>> To: common-user@hadoop.apache.org
>>> Subject: HDFS Backup nodes
>>> 
>>> Does hadoop 0.20.205 supports configuring HDFS backup nodes ?
>>> 
>>> Thanks,
>>> Praveenesh
>>> 
> 
> 


Mime
View raw message