flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kostas Kloudas (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-4939) GenericWriteAheadSink: Decouple the subtask that created a pending checkpoint from the one that commits it.
Date Thu, 27 Oct 2016 08:22:58 GMT
Kostas Kloudas created FLINK-4939:
-------------------------------------

             Summary: GenericWriteAheadSink: Decouple the subtask that created a pending checkpoint
from the one that commits it.
                 Key: FLINK-4939
                 URL: https://issues.apache.org/jira/browse/FLINK-4939
             Project: Flink
          Issue Type: Improvement
          Components: Cassandra Connector
            Reporter: Kostas Kloudas
            Assignee: Kostas Kloudas
             Fix For: 1.2.0


So far the GenericWriteAheadSink expected that
the subtask that wrote a pending checkpoint to the 
state backend, will be also the one to commit it to
the third-party storage system.

This issue targets at removing this assumption. To do this 
the CheckpointCommitter has to be able to dynamically
take the subtaskIdx as a parameter when asking 
if a checkpoint was committed and also change the
state kept by the GenericWriteAheadSink to also 
include that subtask index of the subtask that wrote 
the pending checkpoint.

This change is also necessary for making the operator rescalable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message