hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Shih" <ryan.s...@gmail.com>
Subject Multiple master nodes
Date Tue, 29 Jul 2008 17:54:22 GMT
Dear Hadoop Community --

I am wondering if it is already possible or in the plans to add capability
for multiple master nodes. I'm in a situation where I have a master node
that may potentially be in a less than ideal execution and networking
environment. For this reason, it's possible that the master node could die
at any time. On the other hand, the application must always be available. I
have accessible to me other machines but I'm still unclear on the best
method to add reliability.

Here are a few options that I'm exploring:
a) To create a completely secondary Hadoop cluster that we can flip to when
we detect that the master node has died. This will double hardware costs, so
if we originally have a 5 node cluster, then we would need to pull 5 more
machines out of somewhere for this decision. This is not the preferable
choice.
b) Just mirror the master node via other always available software, such as
DRBD for real time synchronization. Upon detection we could swap to the
alternate node.
c) Or if Hadoop had some functionality already in place, it would be
fantastic to be able to take advantage of that. I don't know if anything
like this is available but I could not find anything as of yet. It seems to
me, however, that having multiple master nodes would be the direction Hadoop
needs to go if it is to be useful in high availability applications. I was
told there are some papers on Amazon's Elastic Computing that I'm about to
look for that follow this approach.

In any case, could someone with experience in solving this type of problem
share how they approached this issue?

Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message