flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Job scheduling
Date Thu, 18 Sep 2014 07:48:47 GMT
Are you referring to this project?
https://github.com/TU-Berlin/dopa-scheduler
Its not an official repository of the Flink (Stratosphere) project. I think
a PhD student at TU Berlin created the code there.



On Thu, Sep 11, 2014 at 4:29 PM, Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> Of course with Flink I could in principle execute almost everything with a
> single Job but, in general, I could write 2 different jobs and decide from
> time to time when the second should be run.
> That's why also metheor scripts are very useful :)
> From what I know there was a scheduler in Stratosphere that was using
> RabbitMQ, right?
>
> I would like to avoid to run linux commands and instead use some REST
> interface to trigger or schedule jobs.
>
> Best,
> Flavio
>
>
> On Thu, Sep 11, 2014 at 4:07 PM, Fabian Hueske <fhueske@apache.org> wrote:
>
>> Hi Flavio,
>>
>> what exactly do you mean by scheduling?
>> Do you want to run a job in regular intervals or execute a complex
>> workflow?
>>
>> Oozie is primarily used to orchestrate the execution of MapReduce
>> workflows. Since, MR is a rather inflexible programming model, complex
>> tasks need to split up into multiple dependent jobs that are executed once
>> their predecessors have finished. Oozie orchestrates this execution.
>> In Flink, you can build a complex analysis flow as a single program and
>> execute it. Hence, there is no need for a workflow scheduler such as Oozie.
>>
>> If you want to run a job in regular intervals, you can configure a cron
>> job, that starts executes the CLI client or implement a Java or Scala
>> program that submits jobs a certain points in time.
>>
>> Best, Fabian
>>
>> 2014-09-11 15:36 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:
>>
>>> Hi to all,
>>>
>>> I'd like to know if there's an example of how to schedule a Job in Flink.
>>> Do we still need something like Oozie or Quartz or we can avoid them?
>>>
>>> Best,
>>> Flavio
>>>
>>
>

Mime
View raw message