hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mapred Learn <mapred.le...@gmail.com>
Subject Re: A way to monitor HDFS for a file to come live, and then kick off a job?
Date Fri, 25 Mar 2011 16:28:13 GMT
Does Oozie co-ordinator work ? Last time I tried it, it had lot of problems:

i) job from start to end_timestamp were all being submitted at once not
at actual wall clock time.

ii) The links to all the jobs in a particular co-ordinator work-flow were
not working i.e. you were not able to see the progress of the jobs running.


On Fri, Mar 25, 2011 at 7:25 AM, Bai, Gang <dev@baigang.net> wrote:

> Hi Jon,
> Oozie could handle this nicely. You may just specify a Oozie coordinator
> jobs. But if you don't have a Oozie server handy, cron jobs could also meet
> your needs.
> Regards,
> -BaiGang
> On Fri, Mar 25, 2011 at 1:09 AM, Jonathan Coveney <jcoveney@gmail.com>wrote:
>> I am not sure if this is the right listserv, forgive me if it is not. My
>> goal is this: monitor HDFS until a file is create, and then kick off a job.
>> Ideally I'd want to do this continuously, but the file would be create
>> hourly (with some sort of variance). I guess I could make a script that
>> would ping the server every 5 minutes or something, but I was wondering if
>> there might be a more elegant way?
>> Thanks
>> Jon

View raw message