From user-return-25280-archive-asf-public=cust-asf.ponee.io@flink.apache.org Fri Jan 11 15:28:56 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id CDE4D180648 for ; Fri, 11 Jan 2019 15:28:55 +0100 (CET) Received: (qmail 61925 invoked by uid 500); 11 Jan 2019 14:28:54 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 61915 invoked by uid 99); 11 Jan 2019 14:28:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jan 2019 14:28:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 4E963C298D for ; Fri, 11 Jan 2019 14:28:54 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.583 X-Spam-Level: ** X-Spam-Status: No, score=2.583 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_IMAGE_ONLY_28=0.726, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=hivehome-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 45tZWVlp5o5h for ; Fri, 11 Jan 2019 14:28:52 +0000 (UTC) Received: from mail-it1-f200.google.com (mail-it1-f200.google.com [209.85.166.200]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5BEDA5F20B for ; Fri, 11 Jan 2019 14:28:52 +0000 (UTC) Received: by mail-it1-f200.google.com with SMTP id x82so2337191ita.9 for ; Fri, 11 Jan 2019 06:28:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hivehome-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=mON8lgmQDM31FL9/oKkIQhZ4Er4pzv2K1o4nMpukLPM=; b=sCQK/zOBqGeY9dg5j2AMvo/ypxioSVQ0ketksV/lD/5vlPzSOqja4cwAHU3cuW6gei tn9p0UslWjABI6povXYBk8W0+bzBAyxtBCRF5To2J95W9C19Z+rpSo8tHFk0PMwfRwu2 lkhLjdiZI8PeiKAo8hyWJn1hKkaZSPXJkYkt1Fr+WVIJjHlgC3aESUAx4VKHSuAvAX3R ErcFoQz+5sxXeGA/qF0q3L2LGy4z67xarVOqHmgNnAHvG4m3bSPJwEZKvbQ5Yqho97gy hYsXBWgNGGA/d0+TgE4VGIJYRWGGMsiD9Oyqhv8Gb64gLJFnPq2nBeDHYDJT9CPa9koo MHoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=mON8lgmQDM31FL9/oKkIQhZ4Er4pzv2K1o4nMpukLPM=; b=c7XpOjefxhWTj7tXaXkyJWVb+klZQEoi4knatt3XIgZfRmQVTs0FEgbOzihvMiegL8 aCOPFzk8IQdwXyRSvNJYIXQ7X0X6Q448MNJ1KeYel1X2hDiCj6rcCAjb+1Vs4i2UWkrZ U1dAFwCwU1VswDKgOPZZ8pROphJK+ifre5YUV811t6PI78hJjtUiGZ5TsEFh5DmVclv6 z4bagz/L3ZZxl2wSiYE3ITeOD7MC89ZUSKk73q5I7Su7FM96H24W+qOuyaLWN4sQxmoJ ZjtuMIl0Sl9cWBIaaJmDXojeUkMU2WD4/PA+LC9A/28vOYx0J7tuxLd1EfvRgiKVfffP nxlg== X-Gm-Message-State: AJcUukfJiHJ1RE+JU9Qz1gZ/ViL1Dps23DjgypXbcSDnj9nMwcmjiVZK Ob3X+f8PxgaOQWE6wqzfAvUTopUJRlR44dFUkm28cqiMMl85hxGTr+TeDXpcgwWA70VLLdKNefB U3PLSMMvMeyFSONV9/Cquss3ENABAWQ== X-Google-Smtp-Source: ALg8bN5RK1aMMLC+muJwy7zdIjMKMzFKVhBJav80M8RVbFy2/A3oyKa8W2FHw22O2eX6NQf1qzcpsPXb+CFqoRre+LY= X-Received: by 2002:a6b:bf41:: with SMTP id p62mr10319852iof.193.1547216931400; Fri, 11 Jan 2019 06:28:51 -0800 (PST) MIME-Version: 1.0 From: Francisco Blaya Date: Fri, 11 Jan 2019 14:28:40 +0000 Message-ID: Subject: Flink 1.7.1 - Parallel tasks in different Task Managers To: user Content-Type: multipart/alternative; boundary="000000000000965e47057f2f83d0" --000000000000965e47057f2f83d0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, We are trying to upgrade to Flink 1.7.1. We have a job with parallelism > 1 over 2 tasks. Due to implementation details outside our control one of these tasks cannot share the same JVM with another parallel instance i.e. if the parallelism is 4 we need 4 slots across different Task Managers. That requirement was met in Flink 1.6.1. Upfront provisioning of Task Managers (8 slots each) and deploying that job first think into the cluster did the trick. However in Flink 1.7.1 it looks like Task Managers are drawn on demand as you deploy jobs. As a consequence we cannot get the parallel tasks deployed to different Task Managers. We have thought of having a separate cluster just for that job where we can configure the Task Managers with one slot each. This way we'll get one parallel instance per Task Manager although we haven't tried it out. Can anyone think of a way of making that work without requiring a separate cluster for that single job? Regards, Fran --=20 hivehome.com Hive | London | Cambridge |=20 Houston | Toronto**** The information contained in or attached to this=20 email is confidential and intended only for the use of the individual(s) to= =20 which it is addressed. It may contain information which is confidential=20 and/or covered by legal professional or other privilege. The views=20 expressed in this email are not necessarily the views of Centrica plc, and= =20 the company, its directors, officers or employees make no representation or= =20 accept any liability for their accuracy or completeness unless expressly=20 stated to the contrary.=C2=A0**** Centrica Hive Limited (company no: 5782908),=20 registered in England and Wales with its registered office at Millstream,= =20 Maidenhead Road, Windsor, Berkshire SL4 5GD. --000000000000965e47057f2f83d0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

We are trying to upgrade to Flink 1= .7.1. We have a job with parallelism > 1 over 2 tasks. Due to implementa= tion details outside our control one of these tasks cannot share the same J= VM with another parallel instance i.e. if the parallelism is 4 we need 4 sl= ots across different Task Managers.

That requireme= nt was met in Flink 1.6.1.=C2=A0Upfront provisioning of Task Managers (8 sl= ots each) and deploying that job first think into the cluster did the trick= .

However in Flink 1.7.1 it looks like Task Manage= rs are drawn on demand as you deploy jobs. As a consequence we cannot get t= he parallel tasks deployed to different Task Managers.

=
We have thought of having a separate cluster just for that job where w= e can configure the Task Managers with one slot each. This way we'll ge= t one parallel instance per Task Manager although we haven't tried it o= ut.

Can anyone think of a way of making that work = without requiring a separate cluster for that single job?

Regards,
Fran

hivehome.com
=



Hive | London | C= ambridge | Houston | Toronto
The information co= ntained in or attached to this email is confidential and intended only for = the use of the individual(s) to which it is addressed. It may contain infor= mation which is confidential and/or covered by legal professional or other = privilege. The views expressed in this email are not necessarily the views = of Centrica plc, and the company, its directors, officers or employees make= no representation or accept any liability for their accuracy or completene= ss unless expressly stated to the contrary.=C2=A0
Centrica Hive Limited (company no: 5782908), registered in England and W= ales with its registered office at Millstream, Maidenhead Road, Windsor, Be= rkshire SL4 5GD.
--000000000000965e47057f2f83d0--