hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Thinking about retriving DFS metadata from datanodes!!!
Date Fri, 12 Sep 2008 21:24:42 GMT
叶双明 wrote:
> Thanks for paying attention  to my tentative idea!
> 
> What I thought isn't how to store the meradata, but the final (or last) way
> to recover valuable data in the cluster when something worst (which destroy
> the metadata in all multiple NameNode) happen. i.e. terrorist attack  or
> natural disasters destroy half of cluster nodes within all NameNode, we can
> recover as much data as possible by this mechanism, and hava big chance to
> recover entire data of cluster because fo original replication.


You want to survive any event that loses a datacentre, you need to 
mirror the data off site, chosing that second site with an up to date 
fault line map of the city, geological knowledge of where recent 
eruptions ended up etc. Which is why nobody builds datacentres in 
Enumclaw WA that I'm aware of, the spec for the fabs in/near portland is 
they ought to withstand 1-2m of volcanic ash landing on them (what 
they'd have got if there'd been an easterly wind when Mount Saint Helens 
went). Then once you have some safe location for the second site, talk 
to your telco about how the high-bandwidth backbones in your city flow 
(Metropolitan Area Ethernet and the like), and try and find somewhere 
that meets your requirements.

Then: come up with a protocol that efficiently keeps the two sites up to 
date. And reliably: S3 went down last month because they'd been using a 
Gossip-style update protocol but wheren't checksumming everything, 
because there's no need on a LAN, but of course on a cross-city network 
more things can go wrong, and for them it did.

Something to keep multiple hadoop filesystems synchronised efficiently 
and reliably across sites could be very useful to many people.

-steve

Mime
View raw message