hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@yahoo-inc.com>
Subject Re: Map Reduce in heterogeneous environ..
Date Thu, 11 Mar 2010 18:44:08 GMT
Another more dynamic way (what I did sometimes back at Sun) is to use JINI
auto-discovery/registry capabilities to keep a central controller with
up-to-date information about registered services configurations.

Worked great. And I agree: pushing configs is messy and error-prone.
  Cos

On Thu, Mar 11, 2010 at 05:45AM, Allen Wittenauer wrote:
> On 3/11/10 4:53 AM, "Segel, Mike" <msegel@navteq.com> wrote:
> > Then push them out respective of their machine types?
> 
> 
> ... and this is the fundamental problem.
> 
> I'm still a big fan of using LDAP to help us solve this problem.  Rather
> than push configs, read them from an LDAP store.  Roughly:
> 
> cn=mygrid
>  .. define common configs
> 
> cn=datanode1,cn=mygrid
> ... datanode properties
> hosts: hostname1,hostname2, @netgroup1, ...   Lists of hosts/netgroups this
> set of properties applies to
> 
> 
> cn=datanode2,
> ... datanode properties
> hosts:hostname4, hostname5, @netgroup2 ... 2nd set of hosts
> 
> Etc.
> 
> Pushing this is part of the problem. If Hadoop was smart enough to read its
> configs from a central store such as LDAP, it would make things much much
> much easier.
> 
> See https://issues.apache.org/jira/browse/HADOOP-5670 for more discussion.
> 

Mime
View raw message