From user-return-25321-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Jan 14 16:21:23 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8404B180608 for ; Mon, 14 Jan 2019 16:21:22 +0100 (CET) Received: (qmail 54079 invoked by uid 500); 14 Jan 2019 15:21:21 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 54050 invoked by uid 99); 14 Jan 2019 15:21:21 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jan 2019 15:21:21 +0000 Received: from [10.0.1.20] (dslb-002-205-086-134.002.205.pools.vodafone-ip.de [2.205.86.134]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id A82A043EF; Mon, 14 Jan 2019 15:21:19 +0000 (UTC) Subject: Re: Parallelism questions To: Alexandru Gutan , User , gary@data-artisans.com, Till Rohrmann References: From: Dawid Wysakowicz Openpgp: preference=signencrypt Autocrypt: addr=dwysakowicz@apache.org; keydata= mQINBFlDn4sBEADAXnPD90xKDWhCifMUi7CQhC2x6YAAnkIAosTqtJiwKQWxD6H/gJof8h6u WryDyvg9aQa6svDYHjnhFlJ15QNUZM6ssqfwtsFx/wPn4NqkOnEXWyybRzv4++Hdci/3fQTu Ht+r69fnDQN/yDCK22KYLBj5r9aqX/M8L3pehulJXHSM/2A5FR5UocWgOh+pbc5ZXwxr72oF DOo+xG31/sMwYV/rQw+6ab5zdNOcuKPR1vxxMGFmYkP1fn3LEiVAq4TGB3y9tpH3jof0u0oS UNRMZg3lQfJxcyhJk+R2Pu+sg3TYI2mAm0sOwxWtl5nn1qxkK8fNsXA9c4Mjc8QWqVxaIQtz yVAhlNQmcKEP0fQhU6m1f6CADaR23J6MHOswfQUbSvKHjmKSo+IjFAYW4PaofXrZwVJmB4KD rsIVnEMeVdYtR3PmABhf3h8ugFKOxQmBpX/CbcH2ycCehRNQhM1uCBy+IF2OGoINuSLCGpy2 ABt1DVGy3G4hgPzlI2OK3xGs7XJWlnO/1uDC0QeSsgr0QwgPJGY/+dumRbadtDd0eae4+/HY zH7vtQd9gXv6qcqVDZr8yd2fzgfo1JJxc6we+ujVwMN0NQkFw6DK0Rob9T5Ip/JfVfOlbIj8 SgIdZBNEHdtB3HEZRZOCGSYBM/8Opa4qXs+3AG3W4Z3vW49ogwARAQABtClEYXdpZCBXeXNh a293aWN6IDxkd3lzYWtvd2ljekBhcGFjaGUub3JnPokCPQQTAQoAJwUCWUOfqAIbAwUJB4Yf gAULCQgHAwUVCgkICwUWAgMBAAIeAQIXgAAKCRAx0t0Qv8FaLeukD/43Vk9Cc6+W76QeEG5o i0aCm4Z2B7Ddo+2Ha0Df6kp3LLYguMl04sU8B4u9+g64/+DZ+4CXYMBlMlmDFOj+k9OLIQTB rmR6CUsq1JVH91zGI3G/FzRWT1ZT5PAL65EJP+asKbjVZJEQIUokEvlzDWhhrwjMhybomC13 Tz76UWuEGXOx4LHadZsu/eaPfb4hZuXPL7e3/8NT+NSfE2BUSLt91Ez371SweU+8A0eT+iBn YG/gYvw19cmIg8CrC2bklrlfQqtMtxDsVXzzz2Ah13JFzQWRABJwA1oJwkrYsb/UvPU3DoSJ cWgzkhm0IKZIKH/YdgahuxF8pf8sp/rHBmJtqftFWDGtBFdmt4t1RXQpE0ikGC8Xa4fcO1Yq UKGPMWBW4DwSuVGbX0imsDC/D/ItvZ8FcjXyGKnCNanc+cZpKzkJpkxZvRsmU8jczSWhsuL1 3Fm9I39CLQtIKRJ0vCIGYK6jlBlNaYAxMNHY+V8GUjUMUcuA27zjOY/1XaOcAcATBBw3He2s QpHZutnAlxkFBoNw3Fs6UGRiurcZ8BXaFcdbdrNE+vv6a2zZA9zrPYPDGZrgsScC+8R8Jw/w 7+vXEQf/CSlALtbKdNsJm4NOLuU/PRqbSl0XWzi4tlV+vtueFVzfxFn55af5O+PrvGTa6tjr yozCG5oG9qOS5BR/9bkCDQRZQ5+LARAAt98V04resJ+wxz0uc8ME+K5fTJiIlbx4PttrMwqO CHbC6eoCnDlcHZE/iqv6tTcYJz8zrHiktpmmBYytL1x9a0lWtY8UjBy+3tjRal2RFCgoMo3X XC4Jlz8kc/4/p2ZQYFJKk35Y5R/Upngr2s7lHLFXoxWg6T9DMwEOoTl6ZC9JGDliyaA7sX56 LtHxkXJN/kEPspoeQenA14Mn2KakYv8ET3vD2HXWPk3EzkbyU0Wz7WOqkyYqNGMDvjGl+/Wl I3qqjrgvYnh7+WuDBpCsP9N2l3XncAwGiMWayePTsu/lkUemAe0LLR9ur5HU701rV9i3bp9U D2/z77VcN3kV3/KIDpMIPHGEGwFqnSW82KJ81fa/1YvZM245XXIPFfvTjJBq9+BqbP+Nvw6z S9pgrO8oo3Va4cgWL4lTKPOYzSWRLeiWUerFSOGXRu+zfu3zjwhy9hCp/DliPkDvJtSQi0MS Wum57ck8ReyIE6y8KKuYblvKXJVQcxaTckJpnx5g0VQKksGHSahD/C7aILp4rNQNkwjOzxh0 qsoBe8yq76XL97+ve7yALM5MUtLt/5LczkI/2EDuGkvXmtN8OpqCFCbqD8UK28bR3g71tfup 7iU6r02+27GTktRnMUpXQ2/u+xBZhKIEnYY9rHEL7lJj3FFntzdbO55K7qwOkn4Lo3EAEQEA AYkCJQQYAQoADwUCWUOfiwIbDAUJB4YfgAAKCRAx0t0Qv8FaLSzlD/9Xv0PaLcoPCeiKnhhn hQR9aLdatfAWBna+iVSDSkE+2g6XKzn6tBjh5J8SzMNVPfxyUjMCZjwW3kzbO6HOmQowrNYd gU4L7Wtn53kZipy1zRGvZAUwjwqs4KI4GUPIK3qrVdyDPAc8WO8wy4bzLsyMq5k1Q0Ud8RDF R03ABDVkElXIkPhe7tn8T9kvQ6apnsIvNs1IZJqBLtzp47f5bWfuSIdjQgVNnQgseeTg1Y08 dIDgvWfJBAuRrw3hdk3gLn13R8vvUL85jtyrk3/4zmivxrkEsllHwtmdobSE4LmJZu94uilD FcHAT0/Ykjmdj7WdnpYAcTRFxMmVgoStJHD/R6iETOB8doyO+L121xCPxJDWDNI95mIypn+i uVueGFmdDYC3V1nBCwuJ7EvVMdb4LmV2iDS00wLqJIfRcVbHnHiR+IhVVOdg3H4sDbw7kY55 SYLI8Ntf/qppwcL3VpvMxfNlL4Ejjam+fxsfA8Qxtvo8VCCzNoFJ8OpA+M8lMyjcmF29IGwK r1+/YDE5IzAfCmwU85KwBpHGwt+fvz6jPnX7fQlWjAbUknc5Z/yuP0UGg5+MAxJjxZgG1ZwS 2wGsUfwiE296v01fUAGnJ8yluUcPODX8ZxfHSSDUd56Dr/2w+pvx06Rjrd+paL0SsyaXqtqG HF857dd5zmwuhWur/g== Message-ID: <4f3e22e2-c965-0ec4-97c0-72bcdc677fd7@apache.org> Date: Mon, 14 Jan 2019 16:21:17 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ZO2wfcuciZq0K9SPtR70adiYiNStKmqOP" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ZO2wfcuciZq0K9SPtR70adiYiNStKmqOP Content-Type: multipart/mixed; boundary="tbhEfhR0xZSaQh8DFpi5d38NmOzwUJhom"; protected-headers="v1" From: Dawid Wysakowicz To: Alexandru Gutan , User , gary@data-artisans.com, Till Rohrmann Message-ID: <4f3e22e2-c965-0ec4-97c0-72bcdc677fd7@apache.org> Subject: Re: Parallelism questions References: In-Reply-To: --tbhEfhR0xZSaQh8DFpi5d38NmOzwUJhom Content-Type: multipart/alternative; boundary="------------38036FCA97D358CA4F2ED3F5" Content-Language: en-US This is a multi-part message in MIME format. --------------38036FCA97D358CA4F2ED3F5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Alexandru As for 2, generally speaking the number of required slots depends on number of slot sharing groups. By default all operators belong to the default slot sharing group, that means a job requires as many slots as maximal parallelism in the job. More on the distributed runtime you can read here[1] As for 1 I cc'ed Gary and Till who might better answer your question. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/concepts/runt= ime.html#task-slots-and-resources Best, Dawid On 14/01/2019 15:26, Alexandru Gutan wrote: > Hi everyone! > > 1. Is there a way to increase the parallelism (e.g. through REST) of > some operators in a job without re-deploying the job? I found this > > answer which mentions scaling at runtime on Yarn/Mesos. Is it > possible? How? Support for Kubernetes? > 2. What happens when the number of parallel operator instances exceeds > the number of task slots? For example: a job with a source > (parallelism 3), a map (parallelism 8), a sink (parallelism 3), total > of *14* operator instances and a setup with *8* task slots. Will the > operators get chained? What if I disable operator chaining? > > Thank you! --------------38036FCA97D358CA4F2ED3F5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Hi Alexandru

As for 2, generally speaking the number of required slots depends on number of slot sharing groups. By default all operators belong to the default slot sharing group, that means a job requires as many slots as maximal parallelism in the job. More on the distributed runtime you can read here[1]

As for 1 I cc'ed Gary and Till who might better answer your question.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/concepts= /runtime.html#task-slots-and-resources

Best,

Dawid

On 14/01/2019 15:26, Alexandru Gutan wrote:
Hi everyone!

1. Is there a way to increase the parallelism (e.g. through REST) of some operators in a job without re-deploying the job? I found this answer which mentions scaling at runtime on Yarn/Mesos. Is it possible? How? Support for Kubernetes?
2. What happens when the number of parallel operator instances exceeds the number of task slots? For example: a job with a source (parallelism 3), a map (parallelism 8), a sink (parallelism 3), total of 14 operator instances and a setup with 8 task slots. Will the operators get chained? What if I disable operator chaining?

Thank you!
--------------38036FCA97D358CA4F2ED3F5-- --tbhEfhR0xZSaQh8DFpi5d38NmOzwUJhom-- --ZO2wfcuciZq0K9SPtR70adiYiNStKmqOP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE6pOkNbTiybTJ9TP2MdLdEL/BWi0FAlw8qO4ACgkQMdLdEL/B Wi2mkQ/+KkINBnwRp4S1VYpDgsT+9zBrwuvvBp/4E1MwFat+pRQAQ9ACrq1BJ7fo jE8xzDaLV5XCPt9Pgcj4JwloMJygBVHyt5ZRyNARoMyuwl4ycTUiLv2JM0vfhavF vUTBmpofdfEYVPJIjpgxl5m2lVkUEc4ba2koNWW6FK38a9jBHETTHAvmiVHtBlYc qb4ZvNLNjp51QCCjJBjqVN66D1RmfF+ES3VBvY4++b3Ui+rzvYPffLRRqDN1Le0X d5SPI5i4ymhy4UWtAYNpY5zA3hEZP83KnuAfFPDUhXyQav72t/kI4i4LgFlZXP5M Hra8cTj8wcdADWdHY/FrbDEpMBBxMKfm/OaObVI4W98WQe5/X5vnc5bipMuhC44Y P/T/+Iq3FMxCfYpI81PF5udX+WlRVOvYujXOqPyNdKiPy9Q0H+tYQ41YDctsNPvZ 3RU94GFge3NXdG2NUq4axMMJxK9IS1gNiCNh2KI1CKhfI1ogSQGJVQvDe25x2KlS +URTDec4g9OZvmdoehiwskmgSF82HNTaJB0JF4Zbh20QsdsSntQfpaJ578Wnne5Q BKcryrf+nEVGMGrFFwarj/7UMgeJuIW9oelH1x0xkKFgjrEyXoYLI8+4Xye4Sdxf +eIfbfZWoumEr/USqMv69X0+s8dKqFO0V1P5hBTeQKTCcq2ITCE= =yBW6 -----END PGP SIGNATURE----- --ZO2wfcuciZq0K9SPtR70adiYiNStKmqOP--