flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Job scheduling
Date Thu, 18 Sep 2014 08:35:36 GMT
I think it could be a useful feature to implement in Stockholm if I could
be there.. :)

On Thu, Sep 18, 2014 at 10:23 AM, Robert Metzger <rmetzger@apache.org>
wrote:

> I don't think that we have a suggested way.
>
> If I would have the requirement, I would look into Oozie. I think its
> quite easy to add additional services (=Flink) into Oozie. In addition, it
> seems to have a REST interface and some other stuff.
>
> If you want, you could also implement one yourself and contribute it back
> to Flink.
>
> On Thu, Sep 18, 2014 at 10:11 AM, Flavio Pompermaier <pompermaier@okkam.it
> > wrote:
>
>> Yes I was referring exactly to that, I was also involved in the Dopa
>> project :)
>> So, at the moment what is the suggested way to schedule jobs with Flink?
>>
>>
>> On Thu, Sep 18, 2014 at 9:48 AM, Robert Metzger <rmetzger@apache.org>
>> wrote:
>>
>>> Are you referring to this project?
>>> https://github.com/TU-Berlin/dopa-scheduler
>>> Its not an official repository of the Flink (Stratosphere) project. I
>>> think a PhD student at TU Berlin created the code there.
>>>
>>>
>>>
>>> On Thu, Sep 11, 2014 at 4:29 PM, Flavio Pompermaier <
>>> pompermaier@okkam.it> wrote:
>>>
>>>> Of course with Flink I could in principle execute almost everything
>>>> with a single Job but, in general, I could write 2 different jobs and
>>>> decide from time to time when the second should be run.
>>>> That's why also metheor scripts are very useful :)
>>>> From what I know there was a scheduler in Stratosphere that was using
>>>> RabbitMQ, right?
>>>>
>>>> I would like to avoid to run linux commands and instead use some REST
>>>> interface to trigger or schedule jobs.
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>>
>>>> On Thu, Sep 11, 2014 at 4:07 PM, Fabian Hueske <fhueske@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Flavio,
>>>>>
>>>>> what exactly do you mean by scheduling?
>>>>> Do you want to run a job in regular intervals or execute a complex
>>>>> workflow?
>>>>>
>>>>> Oozie is primarily used to orchestrate the execution of MapReduce
>>>>> workflows. Since, MR is a rather inflexible programming model, complex
>>>>> tasks need to split up into multiple dependent jobs that are executed
once
>>>>> their predecessors have finished. Oozie orchestrates this execution.
>>>>> In Flink, you can build a complex analysis flow as a single program
>>>>> and execute it. Hence, there is no need for a workflow scheduler such
as
>>>>> Oozie.
>>>>>
>>>>> If you want to run a job in regular intervals, you can configure a
>>>>> cron job, that starts executes the CLI client or implement a Java or
Scala
>>>>> program that submits jobs a certain points in time.
>>>>>
>>>>> Best, Fabian
>>>>>
>>>>> 2014-09-11 15:36 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:
>>>>>
>>>>>> Hi to all,
>>>>>>
>>>>>> I'd like to know if there's an example of how to schedule a Job in
>>>>>> Flink.
>>>>>> Do we still need something like Oozie or Quartz or we can avoid them?
>>>>>>
>>>>>> Best,
>>>>>> Flavio
>>>>>>
>>>>>
>>>>
>>

Mime
View raw message