Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 683D5200C4E for ; Fri, 21 Apr 2017 14:33:21 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 669E1160BA2; Fri, 21 Apr 2017 12:33:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 86FD7160B97 for ; Fri, 21 Apr 2017 14:33:20 +0200 (CEST) Received: (qmail 28821 invoked by uid 500); 21 Apr 2017 12:33:19 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 28811 invoked by uid 99); 21 Apr 2017 12:33:19 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Apr 2017 12:33:19 +0000 Received: from aljoschas-mbp.fritz.box (ipservice-092-214-155-234.092.214.pools.vodafone-ip.de [92.214.155.234]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 4BFE21A00C5; Fri, 21 Apr 2017 12:33:18 +0000 (UTC) From: Aljoscha Krettek Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_BA72E2F0-94D9-4550-90D9-E6D878C3185B" Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Flink slots, threads, task, etc Date: Fri, 21 Apr 2017 14:33:15 +0200 In-Reply-To: Cc: pompermaier@okkam.it To: user@flink.apache.org References: <68554073-B54C-4522-B9FF-751CA4179F25@apache.org> X-Mailer: Apple Mail (2.3273) archived-at: Fri, 21 Apr 2017 12:33:21 -0000 --Apple-Mail=_BA72E2F0-94D9-4550-90D9-E6D878C3185B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, there are currently no built-in metrics for InputSplit consumption but I = do see that this could be quite helpful. I think you can have a custom = RichInputFormat that uses metrics to record stuff, though. I think adding built-in metrics should be possible at this point in the = code: = https://github.com/apache/flink/blob/8f3d6d239996c83f7cbd102dc8a85ee626a56= bf5/flink-runtime/src/main/java/org/apache/flink/runtime/operators/DataSou= rceTask.java#L144-L144 = Best, Aljoscha > On 19. Apr 2017, at 09:25, Flavio Pompermaier = wrote: >=20 > Hi Aljoscha, > thanks for the reply, it was not urgent and I was aware of the = FF...btw, congratulations for it, I saw many interesting talks! > Flink community has grown a lot since it was Stratosphere ;) > Just one last question: in many of my use cases it could be helpful to = see how many of the created splits were "consumed" by an = inputFormat/source. > Is it possible to monitor this part somewhere in the dashboards or = with a custom metric? >=20 > On Tue, Apr 18, 2017 at 5:24 PM, Aljoscha Krettek > wrote: > Hi, > sorry for not getting any responses but I think everyone was quite = busy with Flink Forward SF. I=E2=80=99m also no expert on the topic but = I=E2=80=99ll try and give some answers. >=20 > Regarding a Google Doc version, I don=E2=80=99t think that there is = any. You would have to modify the Markdown version we have in the doc. >=20 > For the other answers I=E2=80=99ll reuse an example program that = consists of Source -> Map -> Sink, with chaining disabled and = parallelism 2. We=E2=80=99ll this have three Tasks: Source, Map, and = Sink, with each having two subtasks. Let=E2=80=99s denote the subtasks = by a number in parenthesis so the first subtask for Source is Source(1), = second one is Source(2). I=E2=80=99ll also refer to Source(1) -> Map(1) = -> Sink(1) as a slice of the execution graph since these can be executed = within one slot. >=20 > Regarding 1, I think this is true. However, a single slot can execute = a complete slice of the execution graph where each subtask (from a = different task) would be executed by its own thread. >=20 > Regarding 2.1, Yes, I think it cannot run multiple subtasks of the = same task while it is possible (and in fact done) to execute all the = subtasks of a slide in the same slot. >=20 > Regarding 2.2, This is so to allow executing a pipeline of parallelism = 8 using a cluster that has 8 free slots. Basically, each slice fills one = slot. >=20 > Regarding 3, I don=E2=80=99t really have an answer. >=20 > Regarding 4, Yes, this can get a bit out of hand if you have very long = pipelines. >=20 > Best, > Aljoscha >=20 >> On 11. Apr 2017, at 14:37, Flavio Pompermaier > wrote: >>=20 >> Any feedback here..? >>=20 >> On Wed, Apr 5, 2017 at 7:43 PM, Flavio Pompermaier = > wrote: >> Hi to all, >> I had a very long but useful chat with Fabian and I understood a lot = of concepts that was not clear at all to me. We started from the Flink = runtime documentation page = (https://ci.apache.org/projects/flink/flink-docs-release-1.2/concepts/runt= ime.html = ) but >> I discovered that the terminology is very inconsistent and misleading = along the page... >>=20 >> For example, one of the very first sentences is : >> "Flink chains operator subtasks together into tasks. Each task is = executed by one thread." >> What I first understood was that every operator can be executed only = by a single thread in all the cluster....probably it should be better = "one thread per task slot" (at least).=20 >> Moreover, if I'm not wrong, a Task Slot can execute only 1 subtask = (aka parallel instance) of each task and there's no limit to the number = of subtasks per slot (and this is not highlighted at all in that = document). The only constraint is that they should belong to different = tasks (right?). >>=20 >> If there's a google doc version of that page I could try to rewrite = it down in order to make it easier to understand some parts...however I = still have some more questions: >> Is it correct that a single Task Slot can execute only a single = subtask of each task and that this task is executed by a single thread = within the slot)? >> If it so: >> why at that page there's written "By default, Flink allows subtasks = to share slots even if they are subtasks of different tasks, so long as = they are from the same job"? It seems that it is more common to run = multiple subtasks of the same task (in a slot) than executing different = substasks of different tasks, although this is still permitted...from = what I understood a slot cannot run multiple subtask of the same task at = all! >> and why this constraint? Is there any good reason for that? A subtask = is mapped to 1 thread in the TaskManager, so why a TM with 2 slots can = run 2 subtasks of the same task (in the same JVM) while a TM with 1 slot = cannot (while it can execute an arbitrary number of subtasks of = different tasks)?=20 >> It it is not so, there's no images representing such a situation in = that page... >> Isn't dangerous to allow (potentially) an unlimited number of threads = per TM slot??=20 >> Cheers, >> Flavio >>=20 >>=20 >=20 --Apple-Mail=_BA72E2F0-94D9-4550-90D9-E6D878C3185B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi,
there are currently no built-in metrics = for InputSplit consumption but I do see that this could be quite = helpful. I think you can have a custom RichInputFormat that uses metrics = to record stuff, though.

I think adding built-in metrics should be possible at this = point in the code: https://github.com/apache/flink/blob/8f3d6d239996c83f7cbd102dc8= a85ee626a56bf5/flink-runtime/src/main/java/org/apache/flink/runtime/operat= ors/DataSourceTask.java#L144-L144

Best,
Aljoscha
On = 19. Apr 2017, at 09:25, Flavio Pompermaier <pompermaier@okkam.it> wrote:

Hi Aljoscha,
thanks for the reply, it was not = urgent and I was aware of the FF...btw, congratulations for it, I saw = many interesting talks!
Flink community has grown a = lot since it was Stratosphere ;)
Just one last = question: in many of my use cases it could be helpful to see how many of = the created splits were "consumed" by an inputFormat/source.
Is it possible to monitor this part somewhere in the = dashboards or with a custom metric?

On Tue, Apr 18, 2017 at 5:24 PM, = Aljoscha Krettek <aljoscha@apache.org> wrote:
Hi,
sorry for = not getting any responses but I think everyone was quite busy with Flink = Forward SF. I=E2=80=99m also no expert on the topic but I=E2=80=99ll try = and give some answers.

Regarding a Google Doc version, I don=E2=80=99t think that = there is any. You would have to modify the Markdown version we have in = the doc.

For = the other answers I=E2=80=99ll reuse an example program that consists of = Source -> Map -> Sink, with chaining disabled and parallelism 2. = We=E2=80=99ll this have three Tasks: Source, Map, and Sink, with each = having two subtasks. Let=E2=80=99s denote the subtasks by a number in = parenthesis so the first subtask for Source is Source(1), second one is = Source(2). I=E2=80=99ll also refer to Source(1) -> Map(1) -> = Sink(1) as a slice of the execution graph since these can be executed = within one slot.

Regarding 1, I think this is true. However, a single slot can = execute a complete slice of the execution graph where each subtask (from = a different task) would be executed by its own thread.

Regarding 2.1, Yes, I = think it cannot run multiple subtasks of the same task while it is = possible (and in fact done) to execute all the subtasks of a slide in = the same slot.

Regarding 2.2, This is so to allow executing a pipeline of = parallelism 8 using a cluster that has 8 free slots. Basically, each = slice fills one slot.

Regarding 3, I don=E2=80=99t really have an answer.

Regarding 4, Yes, this = can get a bit out of hand if you have very long pipelines.

Best,
Aljoscha

On 11. = Apr 2017, at 14:37, Flavio Pompermaier <pompermaier@okkam.it> wrote:

Any feedback here..?

On Wed, = Apr 5, 2017 at 7:43 PM, Flavio Pompermaier <pompermaier@okkam.it> wrote:
Hi to all,
I had a very long but useful chat = with Fabian and I understood a lot of concepts that was not clear at all = to me. We started from the Flink runtime documentation page (https://ci.apache.org/projects/flink/flink-docs-release-1.2/concepts/runtime.html) but
I = discovered that the terminology is very inconsistent and misleading = along the page...

For example, one of the very first sentences is :
"Flink chains operator subtasks together into tasks. Each = task is executed by one thread."
What I first = understood was that every operator can be executed only by a single = thread in all the cluster....probably it should be better "one thread = per task slot" (at least). 
Moreover, if I'm = not wrong, a Task Slot can execute only 1 subtask (aka parallel = instance) of each task and there's no limit to the number of subtasks = per slot (and this is not highlighted at all in that document). The only = constraint is that they should belong to different tasks = (right?).

If = there's a google doc version of that page I could try to rewrite it down = in order to make it easier to understand some parts...however I still = have some more questions:
  1. Is it correct that a single Task Slot can execute only a = single subtask of each task and that this task is executed by a single = thread within the slot)?
  2. If it so:
    1. why at that page there's written "By = default, Flink allows subtasks to share slots even if they are subtasks = of different tasks, so long as they are from the same job"? It seems = that it is more common to run multiple subtasks of the same task (in a = slot) than executing different substasks of different tasks, although = this is still permitted...from what I understood a slot cannot run = multiple subtask of the same task at all!
    2. and why = this constraint? Is there any good reason for that? A subtask is mapped = to 1 thread in the TaskManager, so why a TM with 2 slots can run 2 = subtasks of the same task (in the same JVM) while a TM with 1 slot = cannot  (while it can execute an arbitrary number of subtasks of = different tasks)? 
  3. It it is not so, there's = no images representing such a situation in that page...
  4. Isn't dangerous to allow (potentially) an unlimited number of = threads per TM slot?? 
Cheers,
Flavio




= --Apple-Mail=_BA72E2F0-94D9-4550-90D9-E6D878C3185B--