flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Hogan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4419) Batch improvement for supporting dfs as a ResultPartitionType
Date Fri, 26 Aug 2016 16:11:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439327#comment-15439327

Greg Hogan commented on FLINK-4419:

My apologies for being late to this discussion. Are we not able to recover when a downstream
operator fails if the spilled files are written to redundant storage? That would not require
changes to the DataSet API and would not reduce performance.

CC'ing [~StephanEwen] and [~till.rohrmann]

> Batch improvement for supporting dfs as a ResultPartitionType
> -------------------------------------------------------------
>                 Key: FLINK-4419
>                 URL: https://issues.apache.org/jira/browse/FLINK-4419
>             Project: Flink
>          Issue Type: Improvement
>          Components: Batch Connectors and Input/Output Formats
>            Reporter: shuai.xu
>            Assignee: shuai.xu
> This is the root issue to track a improvement for batch, which will enable dfs as a ResultPartitionType,
so that upstream node can exist totally after finished and need not be restarted if downstream
nodes fail.
> Full design is shown in (https://docs.google.com/document/d/15HtCtc9Gk8SyHsAezM7Od1opAHgnxLeHm3VX7A8fa-4/edit#).

This message was sent by Atlassian JIRA

View raw message