hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Per Steffensen <st...@designware.dk>
Subject Re: Timer jobs
Date Thu, 01 Sep 2011 14:28:28 GMT
Thanks for your response. See comments below.

Regards, Per Steffensen

Alejandro Abdelnur skrev:
> [moving common-user@ to BCC]
>
> Oozie is not HA yet. But it would be relatively easy to make it. It was
> designed with that in mind, we even did a prototype.
>   
Ok, so if it isnt HA out-of-the-box I believe Oozie is too big a 
framework for my needs - I dont need all this workflow stuff - just a 
plain simple job trigger that triggers every 5th minute. I guess I will 
try out something smaller like Quartz Scheduler. It also only have 
HA/cluster support through JDBC (JobStore) but I guess I could fairly 
easy make a HDFSFilesJobStore which still hold the properties so that 
Quartz clustering works.

But what I would really like to have is a scheduling framework that is 
HA out-of-the-box. Guess Oozie is not the solution for me. Anyone knows 
about other frameworks?
> Oozie consists of 2 services, a SQL database to store the Oozie jobs state
> and a servlet container where Oozie app proper runs.
>
> The solution for HA for the database, well, it is left to the database. This
> means, you'll have to get an HA DB.
>   
I would really like to avoid having to run a relational database. 
Couldnt I just do the persistence of Oozie jobs state in files on HDFS?
> The solution for HA for the Oozie app is deploying the servlet container
> with the Oozie app in more than one box (2 or 3); and front them by a HTTP
> load-balancer.
>
> The missing part is that the current Oozie lock-service is currently an
> in-memory implementation. This should be replaced with a Zookeeper
> implementation. Zookeeper could run externally or internally in all Oozie
> servers. This is what was prototyped long ago.
>   
Yes but if I have to do ZooKeeper stuff I could just do the scheduler 
myself and make run no all/many boxes. The only hard part about it is 
the "locking" thing that makes sure only one job-triggering happens in 
the entire cluster when only one job-triggering is supposed to happen, 
and that the job-triggering happens no matter how many machines might be 
down.
> Thanks.
>
> Alejandro
>
>
> On Thu, Sep 1, 2011 at 4:14 AM, Ronen Itkin <ronen@taykey.com> wrote:
>
>   
>> If I get you right you are asking about Installing Oozie as Distributed
>> and/or HA cluster?!
>> In that case I am not familiar with an out of the box solution by Oozie.
>> But, I think you can made up a solution of your own, for example:
>> Installing Oozie on two servers on the same partition which will be
>> synchronized by DRBD.
>> You can trigger a "failover" using linux Heartbeat and that way maintain a
>> virtual IP.
>>
>>
>>
>>
>>
>> On Thu, Sep 1, 2011 at 1:59 PM, Per Steffensen <steff@designware.dk>
>> wrote:
>>
>>     
>>> Hi
>>>
>>> Thanks a lot for pointing me to Oozie. I have looked a little bit into
>>> Oozie and it seems like the "component" triggering jobs is called
>>> "Coordinator Application". But I really see nowhere that this Coordinator
>>> Application doesnt just run on a single machine, and that it will
>>>       
>> therefore
>>     
>>> not trigger anything if this machine is down. Can you confirm that the
>>> "Coordinator Application"-role is distributed in a distribued Oozie
>>>       
>> setup,
>>     
>>> so that jobs gets triggered even if one or two machines are down?
>>>
>>> Regards, Per Steffensen
>>>
>>> Ronen Itkin skrev:
>>>
>>>  Hi
>>>       
>>>> Try to use Oozie for job coordination and work flows.
>>>>
>>>>
>>>>
>>>> On Thu, Sep 1, 2011 at 12:30 PM, Per Steffensen <steff@designware.dk>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>         
>>>>> Hi
>>>>>
>>>>> I use hadoop for a MapReduce job in my system. I would like to have the
>>>>> job
>>>>> run very 5th minute. Are there any "distributed" timer job stuff in
>>>>> hadoop?
>>>>> Of course I could setup a timer in an external timer framework (CRON
or
>>>>> something like that) that invokes the MapReduce job. But CRON is only
>>>>> running on one particular machine, so if that machine goes down my job
>>>>> will
>>>>> not be triggered. Then I could setup the timer on all or many machines,
>>>>> but
>>>>> I would not like the job to be run in more than one instance every 5th
>>>>> minute, so then the timer jobs would need to coordinate who is actually
>>>>> starting the job "this time" and all the rest would just have to do
>>>>> nothing.
>>>>> Guess I could come up with a solution to that - e.g. writing some
>>>>>           
>> "lock"
>>     
>>>>> stuff using HDFS files or by using ZooKeeper. But I would really like
>>>>>           
>> if
>>     
>>>>> someone had already solved the problem, and provided some kind of a
>>>>> "distributed timer framework" running in a "cluster", so that I could
>>>>> just
>>>>> register a timer job with the cluster, and then be sure that it is
>>>>> invoked
>>>>> every 5th minute, no matter if one or two particular machines in the
>>>>> cluster
>>>>> is down.
>>>>>
>>>>> Any suggestions are very welcome.
>>>>>
>>>>> Regards, Per Steffensen
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>
>>>>
>>>>
>>>>         
>>>       
>> --
>> *
>> Ronen Itkin*
>> Taykey | www.taykey.com
>>
>>     
>
>   


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message