hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Per Steffensen <st...@designware.dk>
Subject Re: Timer jobs
Date Thu, 01 Sep 2011 10:59:03 GMT

Thanks a lot for pointing me to Oozie. I have looked a little bit into 
Oozie and it seems like the "component" triggering jobs is called 
"Coordinator Application". But I really see nowhere that this 
Coordinator Application doesnt just run on a single machine, and that it 
will therefore not trigger anything if this machine is down. Can you 
confirm that the "Coordinator Application"-role is distributed in a 
distribued Oozie setup, so that jobs gets triggered even if one or two 
machines are down?

Regards, Per Steffensen

Ronen Itkin skrev:
> Hi
> Try to use Oozie for job coordination and work flows.
> On Thu, Sep 1, 2011 at 12:30 PM, Per Steffensen <steff@designware.dk> wrote:
>> Hi
>> I use hadoop for a MapReduce job in my system. I would like to have the job
>> run very 5th minute. Are there any "distributed" timer job stuff in hadoop?
>> Of course I could setup a timer in an external timer framework (CRON or
>> something like that) that invokes the MapReduce job. But CRON is only
>> running on one particular machine, so if that machine goes down my job will
>> not be triggered. Then I could setup the timer on all or many machines, but
>> I would not like the job to be run in more than one instance every 5th
>> minute, so then the timer jobs would need to coordinate who is actually
>> starting the job "this time" and all the rest would just have to do nothing.
>> Guess I could come up with a solution to that - e.g. writing some "lock"
>> stuff using HDFS files or by using ZooKeeper. But I would really like if
>> someone had already solved the problem, and provided some kind of a
>> "distributed timer framework" running in a "cluster", so that I could just
>> register a timer job with the cluster, and then be sure that it is invoked
>> every 5th minute, no matter if one or two particular machines in the cluster
>> is down.
>> Any suggestions are very welcome.
>> Regards, Per Steffensen

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message