flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francisco Blaya <francisco.bl...@hivehome.com>
Subject Flink 1.7.1 - Parallel tasks in different Task Managers
Date Fri, 11 Jan 2019 14:28:40 GMT

We are trying to upgrade to Flink 1.7.1. We have a job with parallelism > 1
over 2 tasks. Due to implementation details outside our control one of
these tasks cannot share the same JVM with another parallel instance i.e.
if the parallelism is 4 we need 4 slots across different Task Managers.

That requirement was met in Flink 1.6.1. Upfront provisioning of Task
Managers (8 slots each) and deploying that job first think into the cluster
did the trick.

However in Flink 1.7.1 it looks like Task Managers are drawn on demand as
you deploy jobs. As a consequence we cannot get the parallel tasks deployed
to different Task Managers.

We have thought of having a separate cluster just for that job where we can
configure the Task Managers with one slot each. This way we'll get one
parallel instance per Task Manager although we haven't tried it out.

Can anyone think of a way of making that work without requiring a separate
cluster for that single job?


hivehome.com <http://www.hivehome.com>

Hive | London | Cambridge | 
Houston | Toronto****
The information contained in or attached to this 
email is confidential and intended only for the use of the individual(s) to 
which it is addressed. It may contain information which is confidential 
and/or covered by legal professional or other privilege. The views 
expressed in this email are not necessarily the views of Centrica plc, and 
the company, its directors, officers or employees make no representation or 
accept any liability for their accuracy or completeness unless expressly 
stated to the contrary. ****
Centrica Hive Limited (company no: 5782908), 
registered in England and Wales with its registered office at Millstream, 
Maidenhead Road, Windsor, Berkshire SL4 5GD.

View raw message