flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Task Parallelism in a Cluster
Date Mon, 30 Nov 2015 18:18:04 GMT

> On 30 Nov 2015, at 17:47, Kashmar, Ali <Ali.Kashmar@emc.com> wrote:
> Do the parallel instances of each task get distributed across the cluster or is it possible
that they all run on the same node?

Yes, slots are requested from all nodes of the cluster. But keep in mind that multiple tasks
(forming a local pipeline) can be scheduled to the same slot (1 slot can hold many tasks).

Have you seen this? https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/job_scheduling.html

> If they can all run on the same node, what happens when that node crashes? Does the job
manager recreate them using the remaining open slots?

What happens: The job manager tries to restart the program with the same parallelism. Thus
if you have enough free slots available in your cluster, this works smoothly (so yes, the
remaining/available slots are used)

With a YARN cluster the task manager containers are restarted automatically. In standalone
mode, you have to take care of this yourself.

Does this help?

– Ufuk

View raw message