Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
From: Hanna Prinz <hanna_prinz@yahoo.de>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_14B0214F-D26D-43DA-B4D7-2FAB1D5BD97C"
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: Speedup of Flink Applications
Message-Id: <7A05C435-792E-4965-9B97-D65BCBF501CC@yahoo.de>
Date: Tue, 3 Jan 2017 12:25:27 +0100
To: user@flink.apache.org
archived-at: Tue, 03 Jan 2017 11:25:50 -0000


--Apple-Mail=_14B0214F-D26D-43DA-B4D7-2FAB1D5BD97C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Happy new year everyone :)

I=E2=80=99m currently working on a paper about Flink. I already got some =
recommendations on general papers with details about Flink, which helped =
me a lot already. But now that I read them, I=E2=80=99m further =
interested is the speedup capabilities, provided by the Flink Framework: =
How =E2=80=9Efar=E2=80=9C can it scale efficiently?

Amdahls law states that a parallelization is only efficient as long as =
the non-parallelizable part of the processing (time for the =
communication between the nodes etc.) doesn=E2=80=99t =E2=80=9Eeat up=E2=80=
=9C the speed gains of parallelization (=3D parallel slowdown).=20
Of course, the communication overhead is mostly caused by the =
implementation, but the frameworks specific solution for the =
communication between the nodes has a reasonable effect as well.

After studying these papers, it looks like, although Flinks performance =
is better in many cases, the possible speedup is equal to the possible =
speedup of Spark.
1. Spark versus Flink - Understanding Performance in Big Data Analytics =
Frameworks | https://hal.inria.fr/hal-01347638/document
 <https://hal.inria.fr/hal-01347638/document>2. Big Data Analytics on =
Cray XC Series DataWarp using Hadoop, Spark and Flink | =
https://cug.org/proceedings/cug2016_proceedings/includes/files/pap141.pdf =
<https://cug.org/proceedings/cug2016_proceedings/includes/files/pap141.pdf=
>
3. Thrill - High-Performance Algorithmic Distributed Batch Data =
Processing with C++ | =
https://panthema.net/2016/0816-Thrill-High-Performance-Algorithmic-Distrib=
uted-Batch-Data-Processing-with-CPP/1608.05634v1.pdf =
<https://panthema.net/2016/0816-Thrill-High-Performance-Algorithmic-Distri=
buted-Batch-Data-Processing-with-CPP/1608.05634v1.pdf>

Does someone have =E2=80=A6
=E2=80=A6 more information (or data) on speedup of Flink applications?=20=

=E2=80=A6 experience (or data) with Flink in an extremely paralellized =
environment?
=E2=80=A6 detailed information on how the nodes communicate, especially =
when they are waiting for task results of one another?

Thank you very much for your time & answers!
Hanna=

--Apple-Mail=_14B0214F-D26D-43DA-B4D7-2FAB1D5BD97C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">Happy new year everyone :)<div class=3D""><br =
class=3D""></div><div class=3D"">I=E2=80=99m currently working on a =
paper about Flink. I already got some recommendations on general papers =
with details about Flink, which helped me a lot already. But now that I =
read them,<b class=3D""> I=E2=80=99m further interested is the speedup =
capabilities, provided&nbsp;by the Flink Framework: How =E2=80=9Efar=E2=80=
=9C can it scale efficiently?</b></div><div class=3D""><br =
class=3D""></div><div class=3D"">Amdahls law states that =
a&nbsp;parallelization&nbsp;is only efficient as long as the =
non-parallelizable part of the processing (time for the communication =
between the nodes etc.) doesn=E2=80=99t =E2=80=9Eeat up=E2=80=9C the =
speed gains of&nbsp;parallelization (=3D parallel =
slowdown).&nbsp;</div><div class=3D"">Of course, the communication =
overhead is mostly caused by the implementation, but the frameworks =
specific solution for the communication between the nodes has a =
reasonable effect as well.</div><div class=3D""><br class=3D""></div><div =
class=3D"">After studying these papers, it looks like, although Flinks =
performance is better in many cases, the possible speedup is equal to =
the possible speedup of Spark.</div><blockquote style=3D"margin: 0 0 0 =
40px; border: none; padding: 0px;" class=3D"">1. Spark versus Flink - =
Understanding Performance in Big Data Analytics Frameworks |&nbsp;<a =
href=3D"https://hal.inria.fr/hal-01347638/document" =
class=3D"">https://hal.inria.fr/hal-01347638/document<br =
class=3D""></a>2.&nbsp;Big Data Analytics on Cray XC Series DataWarp =
using Hadoop, Spark and Flink |&nbsp;<a =
href=3D"https://cug.org/proceedings/cug2016_proceedings/includes/files/pap=
141.pdf" =
class=3D"">https://cug.org/proceedings/cug2016_proceedings/includes/files/=
pap141.pdf</a><div class=3D"">3. Thrill - High-Performance Algorithmic =
Distributed Batch Data Processing with C++ |&nbsp;<a =
href=3D"https://panthema.net/2016/0816-Thrill-High-Performance-Algorithmic=
-Distributed-Batch-Data-Processing-with-CPP/1608.05634v1.pdf" =
class=3D"">https://panthema.net/2016/0816-Thrill-High-Performance-Algorith=
mic-Distributed-Batch-Data-Processing-with-CPP/1608.05634v1.pdf</a></div><=
/blockquote><div class=3D""><br class=3D""></div><div class=3D"">Does =
someone have =E2=80=A6</div><div class=3D"">=E2=80=A6 more information =
(or data) on speedup of Flink applications?&nbsp;</div><div class=3D"">=E2=
=80=A6 experience (or data) with Flink in an extremely paralellized =
environment?</div><div class=3D"">=E2=80=A6 detailed information on how =
the nodes communicate, especially when they are waiting for task results =
of one another?</div><div class=3D""><br class=3D""></div><div =
class=3D"">Thank you very much for your time &amp; answers!</div><div =
class=3D"">Hanna</div></body></html>=

--Apple-Mail=_14B0214F-D26D-43DA-B4D7-2FAB1D5BD97C--