hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: data center aware hadoop?
Date Sat, 22 Sep 2012 08:39:02 GMT
On 11 September 2012 00:29, Sujee Maniyam <sujee@sujee.net> wrote:

> HI devs
> now that hfds HA is is a reality,  how about HDFS spanning multiple
> data centers?  Are there any discussions / work going on in this area?
> It could be a single cluster spanning multiple data centers or having
> a 'standby cluster' in another data center.
> curious, and thanks for your time!
> regards
> Sujee Maniyam
> http://sujee.net

what are your goals here?

   - store 1 of the 3 replicas off-site for (possible) recovery on a site
   - store 2+ replicas on each site for better recovery of site+block
   - be able to back up all of the data to a different site
   - be able to back up some the data to a different site
   - stream the metadata/NN log to a remote site (you could get away with
   that today

Or do you plan to have different data across sites and then run MR jobs
across them? This would be an interesting problem, but its way above the FS.

There's still a lot of work that could be done for single-site failure
tolerance, in particular
-better failure topology awareness, if you run the site on two external
power supplies -as telcos do- then you want at least one copy on each power
-better partition failure awareness -differentiate "loss of rack"
differently from "all the machines on  rack have stopped reporting in",
which is how it is treated today,


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message