hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From C J <c.josh...@yahoo.com>
Subject Re: Multiple hadoop configurations
Date Fri, 10 Dec 2010 08:04:38 GMT
Thanks everyone. I do have the same version of hadoop on both clusters. And 
oozie does seem to be the best option to get what I need.

- C

From: Venkatesh S <svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Sent: Thu, December 9, 2010 8:36:29 PM
Subject: Re: Multiple hadoop configurations

If you are not running the same version, you could use Reverse Class Loader 
trick to be able to launch jobs to multiple clusters.


On 12/10/10 6:46 AM, "Alejandro Abdelnur" <tucu@cloudera.com> wrote:


If both of your clusters run the same version of Hadoop, then -as
Konstantin suggested- you could use Oozie. Oozie does not rely on
local Hadoop configuration files to determine the JT/NN to use, you
specify them in the Oozie workflow application XML for each job.



On Fri, Dec 10, 2010 at 7:01 AM, C J <c.joshi26@yahoo.com> wrote:
> Thanks Konstantin. My issue is more with supporting the multiple clusters . I 
> using quartz for scheduling the jobs (currently it is 2 separate schedulers 
> the 2 clusters).
> If while submitting a job I am able to specify which cluster it should trigger
> the job on (by giving the handle to the appropriate cluster), I think I will 
> able to manage it.
> Thanks,
> Deepika
> ________________________________
> From: Konstantin Boudnik <cos@apache.org>
> To: general@hadoop.apache.org
> Sent: Thu, December 9, 2010 2:24:34 PM
> Subject: Re: Multiple hadoop configurations
> I believe the answer you are looking for is Oozie coordinator, but I
> am not which version of it supports multiple clusters.
> On Thu, Dec 9, 2010 at 14:20, C J <c.joshi26@yahoo.com> wrote:
>> Hi,
>> I have 2 hadoop clusters . Both these clusters have their own set of jobs
>> running. I also have some distcp jobs which copy over data from one cluster 
>> another.
>> I want to be able to control the jobs on both the clusters through one
>> (so I can coordinate the jobs).
>> I am wondering when I need to trigger a job through a scheduler, how can I
> send
>> to one of the 2 clusters.
>> 2 clusters means 2 sets of configuration files.Is there any way to get a
> handle
>> to one of these clusters, by specifying the configuration file name or
>> something?
>> Will appreciate any help or clues.
>> Thanks.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message