hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <ar...@yahoo-inc.com>
Subject Re: How do people keep their client configurations in sync with the remote cluster(s)
Date Thu, 15 May 2008 15:41:22 GMT

On May 15, 2008, at 5:05 AM, Steve Loughran wrote:

> I have a question for users: how do they ensure their client apps  
> have configuration XML file that are kept up to date?
> I know how I do it to date (get the site config off the site team,  
> have my private copy in SVN), but that is too brittle, and  
> diagnosing failures is pretty tricky. All you get is "Failed to  
> Submit Job!" exceptions and local stack traces, from which you have  
> to work backwards to the work.
> I'm thinking of looking at what it would take for a job submitter  
> to ask the tracker for its config data, to get things like the  
> various directory bases from the cluster, instead of being compiled  
> into the client. Then the management problem becomes one of keeping  
> the cluster configuration under control, which is a much easier  
> proposition.

Fair analysis. http://issues.apache.org/jira/browse/HADOOP-3135 is  
quite close, should get in hadoop-0.18 and will help a great deal. It  
does precisely what you proposed: fix the job-submitter to query the  
JT to get the right configs.


View raw message