flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ufuk Celebi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2119) Add ExecutionGraph support for leg-wise scheduling
Date Sat, 30 May 2015 11:12:17 GMT
Ufuk Celebi created FLINK-2119:
----------------------------------

             Summary: Add ExecutionGraph support for leg-wise scheduling
                 Key: FLINK-2119
                 URL: https://issues.apache.org/jira/browse/FLINK-2119
             Project: Flink
          Issue Type: Improvement
          Components: Scheduler
    Affects Versions: master
            Reporter: Ufuk Celebi
            Assignee: Ufuk Celebi


Scheduling currently happens by lazily unrolling the ExecutionGraph from the sources.

1. All sources are scheduled for execution.
2. Their results trigger scheduling and deployment of the receiving tasks (either on the first
available buffer or when all are produced [pipelined vs. blocking exchange]).

For certain batch jobs this can be problematic as many tasks will be running at the same time
and consume task manager resources like executionslots and memory. For these jobs, it is desirable
to schedule the ExecutionGraph in with different strategies.

With respect to the ExecutionGraph, the current limitation is that data availability for a
result always triggers scheduling of the consuming tasks. This needs to be more general to
allow different scheduling strategies.

Consider the following example:

{code}
          [ union ]
         /         \
        /           \
  [ source 1 ]  [ source 2 ]
{code}

Currently, both sources are scheduled concurrently and the "faster" one triggers scheduling
of the union. It is desirable to first allow source 1 to completly produce its result, then
trigger scheduling of source 2, and only then schedule the union.

The required changes in the ExecutionGraph are conceptually straight-forward: instead of going
through the list of result consumers and scheduling them, we need to be able to run a more
general action. For normal operation, this will still schedule the consumer task, but we can
also configure it to kick of the next source etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message