flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Programmatic management of Flink jobs
Date Tue, 09 May 2017 08:36:34 GMT
Hi,
Not if it doesn’t happen to frequently. Otherwise you would spend most of your time restarting
jobs.

Best,
Aljoscha

> On 9. May 2017, at 08:23, Moiz S Jinia <moiz.jinia@gmail.com> wrote:
> 
> Thanks. Do you foresee any problems with canceling/starting the jobs whenever the patterns
change?
> 
> Moiz
> 
> On Wed, May 3, 2017 at 4:57 PM, Aljoscha Krettek <aljoscha@apache.org <mailto:aljoscha@apache.org>>
wrote:
> Hi,
> Yes, this would work even though it requires canceling/restarting the job whenever the
patterns change.
> 
> Best,
> Aljoscha
> 
>> On 3. May 2017, at 12:36, Moiz S Jinia <moiz.jinia@gmail.com <mailto:moiz.jinia@gmail.com>>
wrote:
>> 
>> The kind of program I intend to submit would be one that sets up a StreamExecutionEnvironment,
connects to a stream from a Kafka topic, and uses a PatternStream over the kafka events. I
could have the jar for this program readily available in the "jobmanager.web.upload.dir",
and use the REST API for submitting a program with some configuration params.
>> 
>> Does that sound like it'd work or am I missing something?
>> 
>> Moiz
>> 
>> On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <moiz.jinia@gmail.com <mailto:moiz.jinia@gmail.com>>
wrote:
>> Not sure I understand Operators. What I need is to have a Pattern that starts consuming
from a Kafka stream. And I need the Patterns to come and go.
>> 
>> Another option that comes to mind is this -
>> The Patterns I'll need are well known in advance. Only certain parameters such as
the time duration of the within clause, and maybe certain filter confitions of the where clause
need tweaking. So I could pre-deploy the Patterns (or jobs) and start or stop them (with parameters).
>> 
>> Does that sound feasible?
>> 
>> On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <aljoscha@apache.org <mailto:aljoscha@apache.org>>
wrote:
>> What would the pattern be added to. An existing custom operator?
>> 
>> The REST interface only allows for managing the lifecycle of a job, not modifying
their graph structure. 
>> 
>>> On 3. May 2017, at 11:43, Moiz S Jinia <moiz.jinia@gmail.com <mailto:moiz.jinia@gmail.com>>
wrote:
>>> 
>>> Thanks for the references. Looking at the REST API, would adding new Patterns
not work via this?
>>> 
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs>
>>> 
>>> Moiz
>>> 
>>> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <aljoscha@apache.org <mailto:aljoscha@apache.org>>
wrote:
>>> Hi,
>>> For managing a Job you can either use the bin/flink command-line tool or the
Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right
now. There are, however, some users that implemented this on top of Flink, see for example
RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and
the other input is a control stream that updates the existing patterns.
>>> 
>>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html>
>>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/ <https://techblog.king.com/rbea-scalable-real-time-analytics-king/>
>>> 
>>>> On 3. May 2017, at 10:02, Moiz S Jinia <moiz.jinia@gmail.com <mailto:moiz.jinia@gmail.com>>
wrote:
>>>> 
>>>> Is there an API that allows remotely adding, modifying, and cancelling Flink
jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.
>>>> 
>>>> Whats the best way to go about this? To the end user the Pattern would manifest
as rules that can be updated anytime.
>>>> 
>>>> 
>>>> Moiz
>>> 
>>> 
>> 
>> 
>> 
> 
> 


Mime
View raw message