Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Wed, 8 Feb 2017 19:39:42 +0000 (UTC)
From: "Stephan Ewen (JIRA)" <jira@apache.org>
To: dev@flink.apache.org
Message-ID: <JIRA.13041462.1486582773000.31605.1486582782149@Atlassian.JIRA>
In-Reply-To: <JIRA.13041462.1486582773000@Atlassian.JIRA>
References: <JIRA.13041462.1486582773000@Atlassian.JIRA> <JIRA.13041462.1486582773537@jira-lw-us.apache.org>
Subject: [jira] [Created] (FLINK-5747) Eager Scheduling should deploy all
 Tasks together
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 08 Feb 2017 19:39:47 -0000

Stephan Ewen created FLINK-5747:
-----------------------------------

             Summary: Eager Scheduling should deploy all Tasks together
                 Key: FLINK-5747
                 URL: https://issues.apache.org/jira/browse/FLINK-5747
             Project: Flink
          Issue Type: Bug
          Components: JobManager
    Affects Versions: 1.2.0
            Reporter: Stephan Ewen
            Assignee: Stephan Ewen
             Fix For: 1.3.0


Currently, eager scheduling immediately triggers the scheduling for all vertices and their subtasks in topological order. 

This has two problems:

  - This works only, as long as resource acquisition is "synchronous". With dynamic resource acquisition in FLIP-6, the resources are returned as Futures which may complete out of order. This results in out-of-order (not in topological order) scheduling of tasks which does not work for streaming.

  - Deploying some tasks that depend on other tasks before it is clear that the other tasks have resources as well leads to situations where many deploy/recovery cycles happen before enough resources are available to get the job running fully.

For eager scheduling, we should allocate all resources in one chunk and then deploy once we know that all are available.

As a follow-up, the same should be done per pipelined component in lazy batch scheduling as well. That way we get lazy scheduling across blocking boundaries, and bulk (gang) scheduling in pipelined subgroups.

This also does not apply for efforts of fine grained recovery, where individual tasks request replacement resources.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)