flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim <mfat...@gmail.com>
Subject Re: Task Slots and Heterogeneous Tasks
Date Fri, 15 Apr 2016 18:08:34 GMT
I see. Sharing slots among subtasks makes sense.
So by default a subtask that executes a map function that calls a  high
latency external service is going to be put in a separate slot. Is it
possible to indicate to the Flink that subtasks of a particular operation
can be collocated in a slot, as such subtasks are IO bound and require no
shared memory?

On Fri, Apr 15, 2016 at 5:31 AM, Till Rohrmann <trohrmann@apache.org> wrote:

> Hi Maxim,
>
> concerning your second part of the question: The managed memory of a
> TaskManager is first split among the available slots. Each slot portion of
> the managed memory is again split among all operators which require managed
> memory when a pipeline is executed. In contrast to that, the heap memory is
> shared by all concurrently running tasks.
>
> Cheers,
> Till
>
> On Fri, Apr 15, 2016 at 1:58 PM, Stephan Ewen <sewen@apache.org> wrote:
>
>> Hi!
>>
>> Slots are usually shared between the heavy and non heavy tasks, for that
>> reason.
>> Have a look at these resources:
>> https://ci.apache.org/projects/flink/flink-docs-master/concepts/concepts.html#workers-slots-resources
>>
>> Let us know if you have more questions!
>>
>> Greetings,
>> Stephan
>>
>>
>> On Fri, Apr 15, 2016 at 1:20 AM, Maxim <mfateev@gmail.com> wrote:
>>
>>> I'm trying to understand a behavior of Flink in case of heterogeneous
>>> operations. For example in our pipelines some operation might accumulate
>>> large windows while another performs high latency calls to external
>>> services. Obviously the former needs task slot with a large memory
>>> allocation, while the latter needs no memory but a high degree of
>>> parallelism.
>>>
>>> Is any way to have different slot types and control allocation of
>>> operations to them? May be is there another way to ensure good hardware
>>> utilization?
>>>
>>> Also from the documentation it is not clear if memory of a TaskManager
>>> is shared across all tasks running on it or each task gets its quota. Could
>>> you clarify it?
>>>
>>> Thanks,
>>>
>>> Maxim.
>>>
>>>
>>>
>>
>

Mime
View raw message