flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1709) Add support for programs with slot count exceeding parallelism
Date Thu, 19 Mar 2015 16:54:39 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369693#comment-14369693

ASF GitHub Bot commented on FLINK-1709:

Github user asfgit closed the pull request at:


> Add support for programs with slot count exceeding parallelism
> --------------------------------------------------------------
>                 Key: FLINK-1709
>                 URL: https://issues.apache.org/jira/browse/FLINK-1709
>             Project: Flink
>          Issue Type: Improvement
>    Affects Versions: master
>            Reporter: Ufuk Celebi
> Currently, we can't run programs with higher parallelism than available slots.
> For example, if we currently have a map-reduce program and 4 task slots configured (e.g.
2 task managers with 2 slots per task manager), the map and reduce tasks will be scheduled
with pipelined results and the same parallelism in shared slots. Setting the parallelism to
more than available slots will result in a NoResourcesAvailableException.
> As a first step to support these kinds of programs, we can add initial support for this
when running in batch mode (after https://github.com/apache/flink/pull/471 is merged).
> This is easier than the original pipelined scenario, because the map tasks can be deployed
after each other to produce the blocking result. The blocking result can then be consumed
after all map tasks produced their result. The mechanism in #471 to deploy result receivers
can be used for this and should not need any modifications.

This message was sent by Atlassian JIRA

View raw message