flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zor X.L." <xiao.liu...@qq.com>
Subject How can I cancel a Flink job safely without a special stop message in the stream?
Date Fri, 11 Aug 2017 01:36:02 GMT
Hi,

What we want to do is cancelling the Flink job after all upstream data 
were processed.
We use Kafka as our input and output, and use the SQL capability of 
Table API by the way.

A possible solution is:

  *

    embed a stop message at the tail of upstream

  *

    do what should be done in the Flink Job

  *

    propagate this stop message to downstream untouched after all data
    are processed

  *

    a downstream monitoring program can thus know if all subtasks are
    finished processing all upstream data

  *

    then cancel the job

*What we want to do is canelling the job safely without utilizing this 
kind of stop message.*

*But I find this is hard or inefficient to implement in Flink… is it 
possible?*

P.S. If not utilizing Flink, a possible solution is:

  *

    the upstream program write a stop signal some where after all data
    were written to Kafka.

      o

        the data has a unique index for exactly once semantics

      o

        the signal should include the last data’s index of all partition

  *

    when the job receive the upstream stop signal

      o

        if the last data of a partition is processed, then this
        partition is finished.

      o

        If all partition is finished, the job can be cancelled

​

Mime
View raw message