Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Mime-Version: 1.0
Date: Sat, 22 Apr 2017 10:42:24 -0700
Message-ID: <58fb95a40b92f30000c513dc@polymail.io>
Subject: Resource overallocation
From: "Luis Alves" <lmtjalves@gmail.com>
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative;
 boundary=622578c68218b2daac239fc2da9d97d61aed56468304a1c70b518318731f
archived-at: Sat, 22 Apr 2017 17:42:51 -0000

--622578c68218b2daac239fc2da9d97d61aed56468304a1c70b518318731f
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

Hi,

Regarding the features on Flink that allow to optimize resource usage in th=
e cluster (+ latency, throughput ...), i.e. slot sharing, task chaining, as=
ync i/o and dynamic scaling, I would like to ask the following questions (a=
ll in the stream processing context):=C2=A0

In which cases would someone be interested in having the number of slots in=
 a task manager higher than the number of cpu cores?=C2=A0

In which case should we prefer split a pipeline of tasks over multiple slot=
s (disable slot sharing), instead of increasing the parallelism, in order f=
or an application to keep up with the incoming data rates?=C2=A0

Is it possible that even when using all the features above, the resources r=
eserved for a slot may be higher than the amount of resources that all the =
tasks in the slot require, thus causing us to have resources that are reser=
ved for a slot, but not being used? Is it possible that such problems appea=
r when we have tasks in applications with different latencies (or different=
 parallelisms)? Or even when we are performing multiple aggregations (that =
cannot be optimised using folds or reduces) on the same window?=C2=A0

Thanks in advance,

Lu=C3=ADs Alves
--622578c68218b2daac239fc2da9d97d61aed56468304a1c70b518318731f--