flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Hogan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3164) Spread out scheduling strategy
Date Fri, 11 Dec 2015 20:17:46 GMT
Greg Hogan created FLINK-3164:

             Summary: Spread out scheduling strategy
                 Key: FLINK-3164
                 URL: https://issues.apache.org/jira/browse/FLINK-3164
             Project: Flink
          Issue Type: Improvement
          Components: Distributed Runtime, Java API, Scala API
    Affects Versions: 1.0.0
            Reporter: Greg Hogan

The size of a Flink cluster is bounded by the amount of memory allocated for network buffers.
The all-to-all distribution of data during a network shuffle means that doubling the number
of TaskManager slots quadruples the required number of network buffers.

A Flink job can be configured to execute operators with lower parallelism which reduces the
number of network buffers used across the cluster. Since the Flink scheduler clusters tasks
the number of network buffers to be configured cannot be reduced.

For example, if each TaskManager has 32 slots and the cluster has 32 TaskManagers the maximum
parallelism can be set to 1024. If the preceding operator has a parallelism of 32 then the
TaskManager fan-out is between 1*1024 (tasks evenly distributed) and 32*1024 (executed on
a single TaskManager).

This message was sent by Atlassian JIRA

View raw message