flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Till Rohrmann (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4419) Batch improvement for supporting dfs as a ResultPartitionType
Date Mon, 29 Aug 2016 10:00:29 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15445435#comment-15445435
] 

Till Rohrmann commented on FLINK-4419:
--------------------------------------

At the moment, the blocking results are kept in managed memory and then spilled to local files
if I'm not mistaken. These results will be released once they are consumed. So at the moment,
the data is lost if a TM dies. Adding DFS backed output/input channel will resolve this problem.

> Batch improvement for supporting dfs as a ResultPartitionType
> -------------------------------------------------------------
>
>                 Key: FLINK-4419
>                 URL: https://issues.apache.org/jira/browse/FLINK-4419
>             Project: Flink
>          Issue Type: Improvement
>          Components: Batch Connectors and Input/Output Formats
>            Reporter: shuai.xu
>            Assignee: shuai.xu
>
> This is the root issue to track a improvement for batch, which will enable dfs as a ResultPartitionType,
so that upstream node can exist totally after finished and need not be restarted if downstream
nodes fail.
> Full design is shown in (https://docs.google.com/document/d/15HtCtc9Gk8SyHsAezM7Od1opAHgnxLeHm3VX7A8fa-4/edit#).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message