flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8547) Implement CheckpointBarrierHandler not to spill data for exactly-once based on credit-based flow control
Date Fri, 09 Feb 2018 09:26:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358151#comment-16358151
] 

ASF GitHub Bot commented on FLINK-8547:
---------------------------------------

Github user zhijiangW commented on the issue:

    https://github.com/apache/flink/pull/5400
  
    @pnowojski , thanks for suggestions and I totally agree with that. 
    That abstraction indeed makes the code simple.  I will update the codes ASAP.


> Implement CheckpointBarrierHandler not to spill data for exactly-once based on credit-based
flow control
> --------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-8547
>                 URL: https://issues.apache.org/jira/browse/FLINK-8547
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Network
>    Affects Versions: 1.5.0
>            Reporter: zhijiang
>            Assignee: zhijiang
>            Priority: Major
>
> Currently in exactly-once mode, the {{BarrierBuffer}} would block inputs with barriers
until all inputs have received the barrier for a given checkpoint. To avoid back-pressuring
the input streams which may cause distributed deadlocks, the {{BarrierBuffer}} has to spill
the data in disk files to recycle the buffers for blocked channels.
>  
> Based on credit-based flow control, every channel has exclusive buffers, so it is no
need to spill data for avoiding deadlock. Then we implement a new {{CheckpointBarrierHandler}} for
only buffering the data for blocked channels for better performance.
>  
> And this new {{CheckpointBarrierHandler}} can also be configured to use or not in order
to rollback the original mode for unexpected risks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message