Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MailScanner-NULL-Check: 1211809109.1778@YQeJj+mgog0MT9rWw8k/xg
Message-ID: <483182D9.3080804@apache.org>
Date: Mon, 19 May 2008 14:38:33 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.14 (X11/20080421)
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Re: How do people keep their client configurations in sync with the
 remote cluster(s)
References: <482D4DD2.7090707@apache.org> <C4530A8E.3F6C9%tdunning@veoh.com>
 <b76e3e240805182006wa584441m92025d47dd86fc94@mail.gmail.com>
In-Reply-To: <b76e3e240805182006wa584441m92025d47dd86fc94@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Alejandro Abdelnur wrote:
> A while ago I've opened an issue related to this topic
> 
>   https://issues.apache.org/jira/browse/HADOOP-3287
> 
> My take is a little different, when submitting a job, the clients
> should only send to the jobtracker the configuration they explicitly
> set, then the job tracker would apply the defaults for all the other
> configuration.
> 
> By doing this the cluster admin can modify things at any time and
> changes on default values take effect for all clients without having
> to distribute a new configuration to all clients.
> 
> IMO, this approach was the intended behavior at some point, according
> to the Configuration.write(OutputStream) javadocs ' Writes non-default
> properties in this configuration.'. But as the write method is writing
> default properties this is not happening.

I'll keep an eye on that issue. I think a key problem right now is that 
clients take their config from the configuration file in the core jar, 
and from their own settings, You need to keep the settings in sync 
somehow, and have to take what the core jar provides.


> This approach would also get rid of the separate mechanism (zookeeper,
> svn, etc) to keep clients synchronized as there would be no need to do
> so.

zookeeper and similar are to keep the cluster alive; they shouldnt be 
needed for clients, which should only need some URL of a job tracker to 
talk to.