Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
From: Gabriele Di Bernardo <gabriele.dibernardo@me.com>
Content-type: multipart/alternative;
 boundary="Apple-Mail=_27E518D3-CE13-4E59-A539-A7A76AF0B506"
MIME-version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Latency on Flink
Message-id: <A8ADCEDB-0505-4892-B2E4-EC384B6F3640@me.com>
Date: Mon, 05 Jun 2017 14:42:10 +0200
To: user@flink.apache.org
archived-at: Mon, 05 Jun 2017 12:42:22 -0000


--Apple-Mail=_27E518D3-CE13-4E59-A539-A7A76AF0B506
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hello everyone,

I am a completely newcomer of streaming engines and big data platforms =
but I am considering using Flink for my master thesis and before =
starting using it I am trying to do some kind of evaluation of the =
system. In particular I am interested in observing how the system reacts =
in terms of latency when it receives a big amount of data to process.

I set up a simple application consisting in:
=E2=80=93 a Kafka producer that generates data for a Kafka topic; each =
data message is distinguished by a source id.

=E2=80=93 a Flink consumer app that reads from Kafka and it should apply =
some kind of reduction operator to the received data (e.g. calculate =
MEAN value of the last 1000 elements received). The Flink consumer keeps =
the state of the messages coming from a certain source (not sure if this =
is the more efficient approach though).=20

I run this application on AWS using EMR with a relatively simple =
configuration:
=E2=80=93 EMR Cluster: Master m3.xlarge (4	CPU + 15GiB Memory), 2 =
Core (2 x m3.xlarge )
=E2=80=93 Kafka + Zookeeper running in a m4.xlarge (4CPU + 16GiB =
memory).

I run the expirement with 2 task managers and 4 slots; I also tried to =
play with the number of partitions of the kafka topic but I experienced =
really high-latency with the increase of the number of messages generate =
per seconds by the Kafka producer. With the simple configuration =
described above I experienced really high latency when for example my =
consumer application generates 5000 double values per seconds; and more =
messages are created more the latency increases.=20

I would like to ask you if, even for this super simple experiment, =
should I scale-out my Flink and/or Kafka cluster to observe better =
performance?

If you have time you can check out my simple code at: =
https://github.com/gdibernardo/streaming-engines-benchmark =
<https://github.com/gdibernardo/streaming-engines-benchmark>. If you =
have any suggestions regarding how to improve my experiments I'd love to =
hear from you.

Thank you so much.

Best,


Gabriele=

--Apple-Mail=_27E518D3-CE13-4E59-A539-A7A76AF0B506
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><p class=3D"">Hello everyone,</p><p class=3D"">I am a =
completely newcomer of streaming engines and big data platforms but I am =
considering using Flink for my master thesis and before starting using =
it I am trying to do some kind of evaluation of the system. In =
particular I am interested in observing how the system reacts in terms =
of latency when it receives a big amount of data to process.</p><p =
class=3D"">I set up a simple application consisting in:<br class=3D"">=E2=80=
=93 a Kafka producer that generates data for a Kafka topic; each data =
message is distinguished by a source id.</p><p class=3D"">=E2=80=93 a =
Flink consumer app that reads from Kafka and it should apply some kind =
of reduction operator to the received data (e.g. calculate MEAN value of =
the last 1000 elements received). The Flink consumer keeps the state of =
the messages coming from a certain source (not sure if this is the more =
efficient approach though).&nbsp;</p><p class=3D"">I run this =
application on AWS using EMR with a relatively simple configuration:<br =
class=3D"">=E2=80=93 EMR Cluster: Master m3.xlarge (4	CPU + 15GiB =
Memory), 2 Core (2 x m3.xlarge )<br class=3D"">=E2=80=93 Kafka + =
Zookeeper running in a m4.xlarge (4CPU + 16GiB memory).</p><p class=3D"">I=
 run the expirement with 2 task managers and 4 slots; I also tried to =
play with the number of partitions of the kafka topic but I experienced =
really high-latency with the increase of the number of messages generate =
per seconds by the Kafka producer. With the simple configuration =
described above I experienced really high latency when for example my =
consumer application generates 5000 double values per seconds; and more =
messages are created more the latency increases.&nbsp;</p><p class=3D"">I =
would like to ask you if, even for this super simple experiment, should =
I scale-out my Flink and/or Kafka cluster to observe better =
performance?</p><p class=3D"">If you have time you can check out my =
simple code at:&nbsp;<a =
href=3D"https://github.com/gdibernardo/streaming-engines-benchmark" =
target=3D"_blank" class=3D"">https://github.com/<wbr =
class=3D"">gdibernardo/streaming-engines-<wbr class=3D"">benchmark</a>. =
If you have any suggestions regarding how to improve my experiments I'd =
love to hear from you.</p><p class=3D"">Thank you so much.</p><p =
class=3D"">Best,</p><p class=3D""><br class=3D""></p><p =
class=3D"">Gabriele</p></body></html>=

--Apple-Mail=_27E518D3-CE13-4E59-A539-A7A76AF0B506--