Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DCE18200C17 for ; Fri, 10 Feb 2017 17:06:06 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id DB5DE160B5B; Fri, 10 Feb 2017 16:06:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2B035160B5C for ; Fri, 10 Feb 2017 17:06:06 +0100 (CET) Received: (qmail 26197 invoked by uid 500); 10 Feb 2017 16:06:05 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 26187 invoked by uid 99); 10 Feb 2017 16:06:05 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Feb 2017 16:06:05 +0000 Received: from mail-it0-f47.google.com (mail-it0-f47.google.com [209.85.214.47]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id CFFD11A06A5 for ; Fri, 10 Feb 2017 16:06:04 +0000 (UTC) Received: by mail-it0-f47.google.com with SMTP id r185so35931686ita.0 for ; Fri, 10 Feb 2017 08:06:04 -0800 (PST) X-Gm-Message-State: AIkVDXKhwD1ocK/fAHdBq930HIqZkw2bZdiIO6rIA/IQwtLObLfBMBLNKAxKF4LILFBCjHFMUu4OuqownKejLg== X-Received: by 10.36.127.132 with SMTP id r126mr28906561itc.57.1486742764236; Fri, 10 Feb 2017 08:06:04 -0800 (PST) MIME-Version: 1.0 References: <1949084242.1893234.1486639798522.ref@mail.yahoo.com> <1949084242.1893234.1486639798522@mail.yahoo.com> In-Reply-To: <1949084242.1893234.1486639798522@mail.yahoo.com> From: Aljoscha Krettek Date: Fri, 10 Feb 2017 16:05:53 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: tasks running in parallel beyond configured parallelism/slots To: Antony Mayi , "user@flink.apache.org" Content-Type: multipart/alternative; boundary=001a1147539455bead05482f46d7 archived-at: Fri, 10 Feb 2017 16:06:07 -0000 --001a1147539455bead05482f46d7 Content-Type: text/plain; charset=UTF-8 Hi, Flink operators will not always (in fact almost never) run in a single slot. Mostly the whole parallel sub-slice of a pipeline can run in one slot, so in your case you get three parallel instances for every operator in your topology and then one instance of each operator will sit in a slot. Cheers, Aljoscha On Thu, 9 Feb 2017 at 12:33 Antony Mayi wrote: > Hi, > > I am new to Flink and bit confused about the execution pipeline of my > Flink job. I run it on cluster of three task managers (flink 1.1.2) each > configured with just single slot. I submit my job with parallelism set to 3. > > This is the global plan (low res - just to show the initial forking): > http://pasteboard.co/weyMrFlZl.png > > This is a detail of the front part: http://pasteboard.co/wez3DVvfW.png > > My confusion is how comes all the parallel operations in the second column > (10 operations) are being executed at the same time if there should be > capacity for max of 3 running at once? Also they are all executed mostly on > same node while the others are idle. > > Thanks for anything useful, > Antony. > --001a1147539455bead05482f46d7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,
Flink operators will not always (in fact almost ne= ver) run in a single slot. Mostly the whole parallel sub-slice of a pipelin= e can run in one slot, so in your case you get three parallel instances for= every operator in your topology and then one instance of each operator wil= l sit in a slot.

Cheers,
Aljoscha
<= /div>
On Thu, 9 Feb 2017 at = 12:33 Antony Mayi <antonymayi@ya= hoo.com> wrote:
Hi,

=
I am new to Flink and bit confused about= the execution pipeline of my Flink job. I run it on cluster of three task = managers (flink 1.1.2) each configured with just single slot. I submit my j= ob with parallelism set to 3.

This is the global plan (= low res - just to show the initial forking):=C2=A0http://pasteboard.co/weyMrFlZl.png

This is a detail of the front part:=C2=A0http://pasteboard.co/wez3DVvfW.png

M= y confusion is how comes all the parallel operations in the second column (= 10 operations) are being executed at the same time if there should be capac= ity for max of 3 running at once? Also they are all executed mostly on same= node while the others are idle.

Thanks for anything usef= ul,
Antony.
--001a1147539455bead05482f46d7--