flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luis Alves" <lmtjal...@gmail.com>
Subject Resource overallocation
Date Sat, 22 Apr 2017 17:42:24 GMT

Regarding the features on Flink that allow to optimize resource usage in the cluster (+ latency,
throughput ...), i.e. slot sharing, task chaining, async i/o and dynamic scaling, I would
like to ask the following questions (all in the stream processing context): 

In which cases would someone be interested in having the number of slots in a task manager
higher than the number of cpu cores? 

In which case should we prefer split a pipeline of tasks over multiple slots (disable slot
sharing), instead of increasing the parallelism, in order for an application to keep up with
the incoming data rates? 

Is it possible that even when using all the features above, the resources reserved for a slot
may be higher than the amount of resources that all the tasks in the slot require, thus causing
us to have resources that are reserved for a slot, but not being used? Is it possible that
such problems appear when we have tasks in applications with different latencies (or different
parallelisms)? Or even when we are performing multiple aggregations (that cannot be optimised
using folds or reduces) on the same window? 

Thanks in advance,

Luís Alves
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message