flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhijiang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-8547) Implement CheckpointBarrierHandler not to spill data for exactly-once based on credit-based flow control
Date Fri, 02 Feb 2018 07:43:00 GMT
zhijiang created FLINK-8547:
-------------------------------

             Summary: Implement CheckpointBarrierHandler not to spill data for exactly-once
based on credit-based flow control
                 Key: FLINK-8547
                 URL: https://issues.apache.org/jira/browse/FLINK-8547
             Project: Flink
          Issue Type: Sub-task
          Components: Network
    Affects Versions: 1.5.0
            Reporter: zhijiang
            Assignee: zhijiang


Currently in exactly-once mode, the `BarrierBuffer` would block inputs with barriers until
all inputs have received the barrier for a given checkpoint. To avoid back-pressuring the
input streams which may cause distributed deadlocks, the `BarrierBuffer` has to spill the
data in disk files to recycle the buffers for blocked channels.

 

Based on credit-based flow control, every channel has exclusive buffers, so it is no need
to spill data for avoiding deadlock. Then we implement a new 'CheckpointBarrierHandler` for
only buffering the data for blocked channels for better performance.

 

And this new `CheckpointBarrierHandler` can also be configured to use or not in order to rollback
the original mode for unexpected risks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message