flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Antoine Philippot (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-7883) Stop fetching source before a cancel with savepoint
Date Fri, 20 Oct 2017 10:23:01 GMT
Antoine Philippot created FLINK-7883:
----------------------------------------

             Summary: Stop fetching source before a cancel with savepoint
                 Key: FLINK-7883
                 URL: https://issues.apache.org/jira/browse/FLINK-7883
             Project: Flink
          Issue Type: Improvement
          Components: DataStream API, Kafka Connector, State Backends, Checkpointing
    Affects Versions: 1.3.2, 1.4.0
            Reporter: Antoine Philippot


For a cancel with savepoint command, the JobManager trigger the cancel call once the savepoint
is finished, but during the savepoint execution, kafka source continue to poll new messages
which will not be part of the savepoint and will be replayed on the next application start.

A solution could be to stop fetching the source stream task before triggering the savepoint.

I suggest to add an interface {{StoppableFetchingSourceFunction}} with a method {{stopFetching}}
that existant SourceFunction implementations could implement.

We can add a {{stopFetchingSource` property in {{CheckpointProperties}} and 
 {{CheckpointOptions}} class to pass the desired behaviour from {{JobManager.handleMessage(CancelJobWithSavepoint)}}
to {{SourceStreamTask.triggerCheckpoint}}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message