flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zor X.L." <xiao.liu...@qq.com>
Subject Re: How can I cancel a Flink job safely without a special stop message in the stream?
Date Mon, 14 Aug 2017 09:15:22 GMT
Bump...


在 2017/8/11 9:36, Zor X.L. 写道:
>
> Hi,
>
> What we want to do is cancelling the Flink job after all upstream data 
> were processed.
> We use Kafka as our input and output, and use the SQL capability of 
> Table API by the way.
>
> A possible solution is:
>
>  *
>
>     embed a stop message at the tail of upstream
>
>  *
>
>     do what should be done in the Flink Job
>
>  *
>
>     propagate this stop message to downstream untouched after all data
>     are processed
>
>  *
>
>     a downstream monitoring program can thus know if all subtasks are
>     finished processing all upstream data
>
>  *
>
>     then cancel the job
>
> *What we want to do is canelling the job safely without utilizing this 
> kind of stop message.*
>
> *But I find this is hard or inefficient to implement in Flink… is it 
> possible?*
>
> P.S. If not utilizing Flink, a possible solution is:
>
>  *
>
>     the upstream program write a stop signal some where after all data
>     were written to Kafka.
>
>      o
>
>         the data has a unique index for exactly once semantics
>
>      o
>
>         the signal should include the last data’s index of all partition
>
>  *
>
>     when the job receive the upstream stop signal
>
>      o
>
>         if the last data of a partition is processed, then this
>         partition is finished.
>
>      o
>
>         If all partition is finished, the job can be cancelled
>
> ​


Mime
View raw message