hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkatesh S <sven...@yahoo-inc.com>
Subject Re: Multiple hadoop configurations
Date Fri, 10 Dec 2010 04:36:29 GMT
If you are not running the same version, you could use Reverse Class Loader trick to be able
to launch jobs to multiple clusters.

-Venkatesh


On 12/10/10 6:46 AM, "Alejandro Abdelnur" <tucu@cloudera.com> wrote:

Deepika,

If both of your clusters run the same version of Hadoop, then -as
Konstantin suggested- you could use Oozie. Oozie does not rely on
local Hadoop configuration files to determine the JT/NN to use, you
specify them in the Oozie workflow application XML for each job.

Cheers.

Alejandro

On Fri, Dec 10, 2010 at 7:01 AM, C J <c.joshi26@yahoo.com> wrote:
> Thanks Konstantin. My issue is more with supporting the multiple clusters . I am
> using quartz for scheduling the jobs (currently it is 2 separate schedulers for
> the 2 clusters).
>
>
> If while submitting a job I am able to specify which cluster it should trigger
> the job on (by giving the handle to the appropriate cluster), I think I will be
> able to manage it.
>
> Thanks,
> Deepika
>
>
>
>
> ________________________________
> From: Konstantin Boudnik <cos@apache.org>
> To: general@hadoop.apache.org
> Sent: Thu, December 9, 2010 2:24:34 PM
> Subject: Re: Multiple hadoop configurations
>
> I believe the answer you are looking for is Oozie coordinator, but I
> am not which version of it supports multiple clusters.
>
> On Thu, Dec 9, 2010 at 14:20, C J <c.joshi26@yahoo.com> wrote:
>> Hi,
>>
>> I have 2 hadoop clusters . Both these clusters have their own set of jobs
>> running. I also have some distcp jobs which copy over data from one cluster to
>> another.
>>
>> I want to be able to control the jobs on both the clusters through one
>>scheduler
>> (so I can coordinate the jobs).
>>
>> I am wondering when I need to trigger a job through a scheduler, how can I
> send
>> to one of the 2 clusters.
>>
>>
>> 2 clusters means 2 sets of configuration files.Is there any way to get a
> handle
>> to one of these clusters, by specifying the configuration file name or
>> something?
>>
>> Will appreciate any help or clues.
>>
>> Thanks.
>>
>>
>>
>>
>
>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message