flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xtra Coder <xtraco...@gmail.com>
Subject Alter Flink's execution graph at run-time
Date Tue, 31 May 2016 06:04:14 GMT
Hello,

I'm curious about ability to alter processing of streams in Flink at
run-time.

Potential use-case may look like following:

1. I have a stream already running (i.e. data processing is already
started) in the Flink's cluster

2. At some point of time I decide that I need to add some more steps in the
data-processing graph, for example i'd like to add logging of intermediate
results after some 'flatMap()' function.Roughly this may look like adding
another `flatMap()` into original graph - it just pipes 'in' to 'out'
without modification (thus not affecting logic of 'current' flow and
already generated state) and internally writes payload to log or sends to
Kafka for further processing.

I guess, same way it should be possible to remove 'transformationless'
functions from the execution graph at run-time without stopping the
execution.

Is this possible? or planed to be possible?

Thanks.

Mime
View raw message