flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Antoine Philippot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7883) Stop fetching source before a cancel with savepoint
Date Fri, 20 Oct 2017 14:53:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212748#comment-16212748

Antoine Philippot commented on FLINK-7883:

I made a commit to illustrate the solution https://github.com/aphilippot/flink/commit/9c58c95bb4b68ea337f7c583b7e039d86f3142a6

If someone can validate the idea or comment it, I would be delighted to submit a pull request

> Stop fetching source before a cancel with savepoint
> ---------------------------------------------------
>                 Key: FLINK-7883
>                 URL: https://issues.apache.org/jira/browse/FLINK-7883
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API, Kafka Connector, State Backends, Checkpointing
>    Affects Versions: 1.4.0, 1.3.2
>            Reporter: Antoine Philippot
> For a cancel with savepoint command, the JobManager trigger the cancel call once the
savepoint is finished, but during the savepoint execution, kafka source continue to poll new
messages which will not be part of the savepoint and will be replayed on the next application
> A solution could be to stop fetching the source stream task before triggering the savepoint.
> I suggest to add an interface {{StoppableFetchingSourceFunction}} with a method {{stopFetching}}
that existant SourceFunction implementations could implement.
> We can add a {{stopFetchingSource}} property in 
>  {{CheckpointOptions}} class to pass the desired behaviour from {{JobManager.handleMessage(CancelJobWithSavepoint)}}
to {{SourceStreamTask.triggerCheckpoint}}

This message was sent by Atlassian JIRA

View raw message