flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhu Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10429) Redesign Flink Scheduling, introducing dedicated Scheduler component
Date Thu, 27 Sep 2018 09:23:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630038#comment-16630038

Zhu Zhu commented on FLINK-10429:


Really like this proposal. Having a centered scheduler makes the scheduling and related
events easier to track, which can support the pluggable scheduling the JM failover well.
A late expanded ExecutionGraph is also we've thought long to achieve better resource utilization
based on runtime feedbacks.

Glad we are moving in the same direction. Wish we can together make the scheduling better.


> Redesign Flink Scheduling, introducing dedicated Scheduler component
> --------------------------------------------------------------------
>                 Key: FLINK-10429
>                 URL: https://issues.apache.org/jira/browse/FLINK-10429
>             Project: Flink
>          Issue Type: New Feature
>          Components: Distributed Coordination
>    Affects Versions: 1.7.0
>            Reporter: Stefan Richter
>            Assignee: Stefan Richter
>            Priority: Major
> This epic tracks the redesign of scheduling in Flink. Scheduling is currently a concern
that is scattered across different components, mainly the ExecutionGraph/Execution and the
SlotPool. Scheduling also happens only on the granularity of individual tasks, which make
holistic scheduling strategies hard to implement. In this epic we aim to introduce a dedicated
Scheduler component that can support use-case like auto-scaling, local-recovery, and resource
optimized batch.
> The design for this feature is developed here: https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing

This message was sent by Atlassian JIRA

View raw message