flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: Task Slots and Heterogeneous Tasks
Date Mon, 18 Apr 2016 16:58:14 GMT
No, it's not possible at the moment to deploy more than one task of the
same kind to a single slot.

On Fri, Apr 15, 2016 at 8:08 PM, Maxim <mfateev@gmail.com> wrote:

> I see. Sharing slots among subtasks makes sense.
> So by default a subtask that executes a map function that calls a  high
> latency external service is going to be put in a separate slot. Is it
> possible to indicate to the Flink that subtasks of a particular operation
> can be collocated in a slot, as such subtasks are IO bound and require no
> shared memory?
> On Fri, Apr 15, 2016 at 5:31 AM, Till Rohrmann <trohrmann@apache.org>
> wrote:
>> Hi Maxim,
>> concerning your second part of the question: The managed memory of a
>> TaskManager is first split among the available slots. Each slot portion of
>> the managed memory is again split among all operators which require managed
>> memory when a pipeline is executed. In contrast to that, the heap memory is
>> shared by all concurrently running tasks.
>> Cheers,
>> Till
>> On Fri, Apr 15, 2016 at 1:58 PM, Stephan Ewen <sewen@apache.org> wrote:
>>> Hi!
>>> Slots are usually shared between the heavy and non heavy tasks, for that
>>> reason.
>>> Have a look at these resources:
>>> https://ci.apache.org/projects/flink/flink-docs-master/concepts/concepts.html#workers-slots-resources
>>> Let us know if you have more questions!
>>> Greetings,
>>> Stephan
>>> On Fri, Apr 15, 2016 at 1:20 AM, Maxim <mfateev@gmail.com> wrote:
>>>> I'm trying to understand a behavior of Flink in case of heterogeneous
>>>> operations. For example in our pipelines some operation might accumulate
>>>> large windows while another performs high latency calls to external
>>>> services. Obviously the former needs task slot with a large memory
>>>> allocation, while the latter needs no memory but a high degree of
>>>> parallelism.
>>>> Is any way to have different slot types and control allocation of
>>>> operations to them? May be is there another way to ensure good hardware
>>>> utilization?
>>>> Also from the documentation it is not clear if memory of a TaskManager
>>>> is shared across all tasks running on it or each task gets its quota. Could
>>>> you clarify it?
>>>> Thanks,
>>>> Maxim.

View raw message