hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: HADOOP-4539 question
Date Mon, 21 Sep 2009 04:18:18 GMT
On Sun, Sep 20, 2009 at 7:38 PM, Stas Oskin <stas.oskin@gmail.com> wrote:
> Hi.
>
> Just wanted to find out about the status of this feature.
>
> Any idea what release this is planned for?
>
> Regards.
>
> 2009/8/17 Edward Capriolo <edlinuxguru@gmail.com>
>
>> There are some native ha like solutions that feature clustering
>> electing a dc and messaging. Check out shoal. I tinkered with build a
>> linux ha like kit over shoal.
>>
>> On 8/13/09, Konstantin Shvachko <shv@yahoo-inc.com> wrote:
>> > There is no "native" HA solution for HDFS at the moment.
>> > "External" HA solutions, like Coudera's may exist.
>> > Cannot speak for everybody, but I know at least one different approach.
>> >
>> > --Konstantin
>> >
>> > Stas Oskin wrote:
>> >> Hi.
>> >>
>> >>> This is exactly the goal (long term). To evolve BN into StandbyNode,
>> >>> which will be able to take over when main NN dies without restarting
>> >>> anything else.
>> >>> And the only remaining step is to implement fail-over mechanism.
>> >>>
>> >>>
>> >>
>> >> Just to clarify, for the near future, the only HA option is Cloudera
>>  DRDB
>> >> approach.
>> >>
>> >> Correct?
>> >>
>> >
>>
>

Just for reference. Linux HA and some other tools deal with the split
brain decisions by requiring a quorum. A quorum involves having a
third party or having more then 50% of the nodes agree.

An issue with linux-ha and hadoop is that linux-ha is only
supported/tested on clusters of up to 16 nodes. That is not a hard
limit, but no one claims to have done it on 1000 or so nodes. You
could just install linux HA on a random sampling of 10 nodes across
your network. That would in theory create an effective quorum.

There are other HA approaches that do not involve DRBD. One is store
your name node table on a SAN or and NFS server. Terracotta is another
option that you might want to look at. But no, at the moment there is
no fail-over built into hadoop.

Mime
View raw message