flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: Task Slots and Heterogeneous Tasks
Date Fri, 15 Apr 2016 12:31:37 GMT
Hi Maxim,

concerning your second part of the question: The managed memory of a
TaskManager is first split among the available slots. Each slot portion of
the managed memory is again split among all operators which require managed
memory when a pipeline is executed. In contrast to that, the heap memory is
shared by all concurrently running tasks.

Cheers,
Till

On Fri, Apr 15, 2016 at 1:58 PM, Stephan Ewen <sewen@apache.org> wrote:

> Hi!
>
> Slots are usually shared between the heavy and non heavy tasks, for that
> reason.
> Have a look at these resources:
> https://ci.apache.org/projects/flink/flink-docs-master/concepts/concepts.html#workers-slots-resources
>
> Let us know if you have more questions!
>
> Greetings,
> Stephan
>
>
> On Fri, Apr 15, 2016 at 1:20 AM, Maxim <mfateev@gmail.com> wrote:
>
>> I'm trying to understand a behavior of Flink in case of heterogeneous
>> operations. For example in our pipelines some operation might accumulate
>> large windows while another performs high latency calls to external
>> services. Obviously the former needs task slot with a large memory
>> allocation, while the latter needs no memory but a high degree of
>> parallelism.
>>
>> Is any way to have different slot types and control allocation of
>> operations to them? May be is there another way to ensure good hardware
>> utilization?
>>
>> Also from the documentation it is not clear if memory of a TaskManager is
>> shared across all tasks running on it or each task gets its quota. Could
>> you clarify it?
>>
>> Thanks,
>>
>> Maxim.
>>
>>
>>
>

Mime
View raw message