flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Job scheduling
Date Thu, 18 Sep 2014 08:23:28 GMT
I don't think that we have a suggested way.

If I would have the requirement, I would look into Oozie. I think its quite
easy to add additional services (=Flink) into Oozie. In addition, it seems
to have a REST interface and some other stuff.

If you want, you could also implement one yourself and contribute it back
to Flink.

On Thu, Sep 18, 2014 at 10:11 AM, Flavio Pompermaier <pompermaier@okkam.it>

> Yes I was referring exactly to that, I was also involved in the Dopa
> project :)
> So, at the moment what is the suggested way to schedule jobs with Flink?
> On Thu, Sep 18, 2014 at 9:48 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
>> Are you referring to this project?
>> https://github.com/TU-Berlin/dopa-scheduler
>> Its not an official repository of the Flink (Stratosphere) project. I
>> think a PhD student at TU Berlin created the code there.
>> On Thu, Sep 11, 2014 at 4:29 PM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>> Of course with Flink I could in principle execute almost everything with
>>> a single Job but, in general, I could write 2 different jobs and decide
>>> from time to time when the second should be run.
>>> That's why also metheor scripts are very useful :)
>>> From what I know there was a scheduler in Stratosphere that was using
>>> RabbitMQ, right?
>>> I would like to avoid to run linux commands and instead use some REST
>>> interface to trigger or schedule jobs.
>>> Best,
>>> Flavio
>>> On Thu, Sep 11, 2014 at 4:07 PM, Fabian Hueske <fhueske@apache.org>
>>> wrote:
>>>> Hi Flavio,
>>>> what exactly do you mean by scheduling?
>>>> Do you want to run a job in regular intervals or execute a complex
>>>> workflow?
>>>> Oozie is primarily used to orchestrate the execution of MapReduce
>>>> workflows. Since, MR is a rather inflexible programming model, complex
>>>> tasks need to split up into multiple dependent jobs that are executed once
>>>> their predecessors have finished. Oozie orchestrates this execution.
>>>> In Flink, you can build a complex analysis flow as a single program and
>>>> execute it. Hence, there is no need for a workflow scheduler such as Oozie.
>>>> If you want to run a job in regular intervals, you can configure a cron
>>>> job, that starts executes the CLI client or implement a Java or Scala
>>>> program that submits jobs a certain points in time.
>>>> Best, Fabian
>>>> 2014-09-11 15:36 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:
>>>>> Hi to all,
>>>>> I'd like to know if there's an example of how to schedule a Job in
>>>>> Flink.
>>>>> Do we still need something like Oozie or Quartz or we can avoid them?
>>>>> Best,
>>>>> Flavio

View raw message