flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2119) Add ExecutionGraph support for leg-wise scheduling
Date Thu, 04 Jun 2015 11:08:38 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572542#comment-14572542

ASF GitHub Bot commented on FLINK-2119:

Github user StephanEwen commented on the pull request:

    The hook that starts the next batch source is at the level of the `Execution`. That means
that as soon as one execution is finished, the next leg would start.
    I think that would only work if the leg's does not by itself trigger any multiple successors
(because it has an intermediate shuffle). Otherwise, we would again immediately exceed the
number of slots.
    Can we put the hook onto the intermediate result, and have it trigger an entire "slot
sharing group" as soon as it is fully available? My initial thought is that this would support
the above case better.

> Add ExecutionGraph support for leg-wise scheduling
> --------------------------------------------------
>                 Key: FLINK-2119
>                 URL: https://issues.apache.org/jira/browse/FLINK-2119
>             Project: Flink
>          Issue Type: Improvement
>          Components: Scheduler
>    Affects Versions: master
>            Reporter: Ufuk Celebi
>            Assignee: Ufuk Celebi
> Scheduling currently happens by lazily unrolling the ExecutionGraph from the sources.
> 1. All sources are scheduled for execution.
> 2. Their results trigger scheduling and deployment of the receiving tasks (either on
the first available buffer or when all are produced [pipelined vs. blocking exchange]).
> For certain batch jobs this can be problematic as many tasks will be running at the same
time and consume task manager resources like executionslots and memory. For these jobs, it
is desirable to schedule the ExecutionGraph in with different strategies.
> With respect to the ExecutionGraph, the current limitation is that data availability
for a result always triggers scheduling of the consuming tasks. This needs to be more general
to allow different scheduling strategies.
> Consider the following example:
> {code}
>           [ union ]
>          /         \
>         /           \
>   [ source 1 ]  [ source 2 ]
> {code}
> Currently, both sources are scheduled concurrently and the "faster" one triggers scheduling
of the union. It is desirable to first allow source 1 to completly produce its result, then
trigger scheduling of source 2, and only then schedule the union.
> The required changes in the ExecutionGraph are conceptually straight-forward: instead
of going through the list of result consumers and scheduling them, we need to be able to run
a more general action. For normal operation, this will still schedule the consumer task, but
we can also configure it to kick of the next source etc.

This message was sent by Atlassian JIRA

View raw message