hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: corrupted fsimage and edits
Date Wed, 30 Jul 2008 22:21:51 GMT
Torsten Curdt wrote:
> 
> On Jul 30, 2008, at 20:35, Raghu Angadi wrote:
> 
>> You should always have more than one location (preferably on different 
>> disks) for fsimage and editslog.
> 
> On production we do frequent backups. Is there a mechanism from inside 
> hadoop now to do something like that now? The "more than one location" 
> bit sounds a little like that.

You can specify multiple directories for "dfs.name.dir", in which case 
fsimage and editslog are written to multiple places. If one of these 
goes bad, you can use the other one.

See http://wiki.apache.org/hadoop/FAQ#15

Raghu.

>> A few months back I had a proposal to keep checksums for each record 
>> on fsimage and editslog and NameNode would recover transparently from 
>> such corruptions when there are more than one copies available. It 
>> didn't come up in priority since there were no such failures observed.
>>
>> You should certainly report these cases and will help the feature gain 
>> more traction.
> 
> Will file a bug report tomorrow.
> 
> cheers
> -- 
> Torsten


Mime
View raw message