flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1671) Add execution modes for programs
Date Tue, 10 Mar 2015 12:41:38 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354825#comment-14354825
] 

Robert Metzger commented on FLINK-1671:
---------------------------------------

That sounds pretty awesome. I agree to make the {{PIPELINED}} mode default.

> Add execution modes for programs
> --------------------------------
>
>                 Key: FLINK-1671
>                 URL: https://issues.apache.org/jira/browse/FLINK-1671
>             Project: Flink
>          Issue Type: Bug
>    Affects Versions: 0.9
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 0.9
>
>
> Currently, there is a single way that programs get executed: Pipelined. With the new
code for batch shuffles (https://github.com/apache/flink/pull/471), we have much more flexibility
and I would like to expose that.
> I suggest to add more execution modes that can be chosen on the `ExecutionEnvironment`:
>   - {{BATCH}} A mode where every shuffle is executed in a batch way, meaning preceding
operators must be done before successors start. Only for the batch programs (d'oh).
>   - {{PIPELINED}} This is the mode corresponding to the current execution mode. It pipelines
where possible and batches, where deadlocks would otherwise happen. Initially, I would make
this the default (be close to the current behavior). Only available for batch programs.
>   - {{PIPELINED_WITH_BATCH_FALLBACK}} This would start out with pipelining shuffles and
fall back to batch shuffles upon failure and recovery, or once it sees that not enough slots
are available to bring up all operators at once (requirement for pipelining).
>   - {{STREAMING}} This is the default and only way for streaming programs. All communication
is pipelined, and the special streaming checkpointing code is activated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message