hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?
Date Thu, 06 Oct 2011 09:54:05 GMT
On 06/10/11 06:40, Jagane Sundar wrote:
> On Wed, Oct 5, 2011 at 10:09 PM, Konstantin Boudnik<cos@apache.org>  wrote:
>
>> On Wed, Oct 05, 2011 at 07:00PM, Jagane Sundar wrote:
>>> approaches you are familiar with. Chef/Puppet et. al. are not interesting
>> to
>>
>> Is this a technical lack of interest as in these solutions do not perform
>> as
>> you expect them or this is a policy thing of some kind?
>>
>
> No policy or anything of that sort. It's a personal preference. Chef,
> puppet, etc. are not full feedback systems. They keep doing the same thing
> over and over again trying to to get the system into a 'desired' state. A
> state machine driven full feedback system works better. When things go
> wrong, that information can be acted upon.

You've just started the script vs goal seeking CM war. goal-seeking has 
better recovery, but can jitter between different desired states in the 
classic strange attractor patten. It federates well, and if you look at 
intranet and internet routing, it's roughly what happens there, though 
BGP lets autonomous networks make their own policy decisions.

Scripts work if the starting state is always the same; they put you in 
the same final state. Usually.

for more details, see:
http://www.slideshare.net/steve_l/dynamic-hadoop-clusters

In large clusters you want consistent machine state and that is where 
HPC-style solutions win: they can work with ILO nets to do things like 
BIOS upgrades, and install the RPM set in a strict order to ensure the 
final state is the same. (RPMs contain scripts, remember).

DevOps style tooling is good for dynamic clusters, esp on cloud 
infrastructure -which is the most dynamic, and where you can manage a 
single starting image, which makes it implicitly consistent across all 
nodes.

-steve

Mime
View raw message