hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "叶双明" <yeshuangm...@gmail.com>
Subject Re: Thinking about retriving DFS metadata from datanodes!!!
Date Tue, 16 Sep 2008 00:39:12 GMT
Thanks, Steve Loughran. I learned sth from you!

2008/9/13 Steve Loughran <stevel@apache.org>

> 叶双明 wrote:
>
>> Thanks for paying attention  to my tentative idea!
>>
>> What I thought isn't how to store the meradata, but the final (or last)
>> way
>> to recover valuable data in the cluster when something worst (which
>> destroy
>> the metadata in all multiple NameNode) happen. i.e. terrorist attack  or
>> natural disasters destroy half of cluster nodes within all NameNode, we
>> can
>> recover as much data as possible by this mechanism, and hava big chance to
>> recover entire data of cluster because fo original replication.
>>
>
>
> You want to survive any event that loses a datacentre, you need to mirror
> the data off site, chosing that second site with an up to date fault line
> map of the city, geological knowledge of where recent eruptions ended up
> etc. Which is why nobody builds datacentres in Enumclaw WA that I'm aware
> of, the spec for the fabs in/near portland is they ought to withstand 1-2m
> of volcanic ash landing on them (what they'd have got if there'd been an
> easterly wind when Mount Saint Helens went). Then once you have some safe
> location for the second site, talk to your telco about how the
> high-bandwidth backbones in your city flow (Metropolitan Area Ethernet and
> the like), and try and find somewhere that meets your requirements.
>
> Then: come up with a protocol that efficiently keeps the two sites up to
> date. And reliably: S3 went down last month because they'd been using a
> Gossip-style update protocol but wheren't checksumming everything, because
> there's no need on a LAN, but of course on a cross-city network more things
> can go wrong, and for them it did.
>
> Something to keep multiple hadoop filesystems synchronised efficiently and
> reliably across sites could be very useful to many people.
>
> -steve
>



-- 
Sorry for my english!! 明
Please help me to correct my english expression and error in syntax
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message