hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: How do people keep their client configurations in sync with the remote cluster(s)
Date Mon, 19 May 2008 13:38:33 GMT
Alejandro Abdelnur wrote:
> A while ago I've opened an issue related to this topic
>   https://issues.apache.org/jira/browse/HADOOP-3287
> My take is a little different, when submitting a job, the clients
> should only send to the jobtracker the configuration they explicitly
> set, then the job tracker would apply the defaults for all the other
> configuration.
> By doing this the cluster admin can modify things at any time and
> changes on default values take effect for all clients without having
> to distribute a new configuration to all clients.
> IMO, this approach was the intended behavior at some point, according
> to the Configuration.write(OutputStream) javadocs ' Writes non-default
> properties in this configuration.'. But as the write method is writing
> default properties this is not happening.

I'll keep an eye on that issue. I think a key problem right now is that 
clients take their config from the configuration file in the core jar, 
and from their own settings, You need to keep the settings in sync 
somehow, and have to take what the core jar provides.

> This approach would also get rid of the separate mechanism (zookeeper,
> svn, etc) to keep clients synchronized as there would be no need to do
> so.

zookeeper and similar are to keep the cluster alive; they shouldnt be 
needed for clients, which should only need some URL of a job tracker to 
talk to.

View raw message