Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <D1CE7634.68236%jsparks@cray.com>
References: <D1CE7634.68236%jsparks@cray.com>
Date: Fri, 17 Jul 2015 16:19:11 +0200
Message-ID: 
 <CAAdrtT3bHBtc1-K1tHYLCR=aQXvGoLgvXGTSbMxU7Euf1uaWUw@mail.gmail.com>
Subject: Re: Map and Reduce cycle
From: Fabian Hueske <fhueske@gmail.com>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1134946c31fe48051b12df0d

--001a1134946c31fe48051b12df0d
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Bill,

Flink uses pipelined data shipping by default. If you have a program like:
source -> map -> reduce -> sink, the mapper will immediately start to
shuffle data over the network to the reducer. The reducer collects that
data and starts to sort it in batches. When the mapper is done and the
reducer is done with the sorting, the sorted batches are merged and a
sorted stream is fed into the reduce function.
So, the shuffling and sorting happens while the mapper is running, but the
reduce function can only be applied after the last mapper has finished.

I am not aware of benchmarks that compare Flink's and Spark's in-memory
operations, but this blog post compares the performance of sorting binary
data in managed memory (like Flink) to naive sorting on the heap (Spark
might do it differently!).

-->
http://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html

Fabian


2015-07-17 16:07 GMT+02:00 Bill Sparks <jsparks@cray.com>:

>  Does flink require all the map tasks to finish before the reducers can
> proceed like Spark, or can the reducer operations start before all the
> mappers have finished like the older Hadoop mapreduce.
>
>  Also my understanding is that flink manages it's own heap, do you/we
> have a sense of the performance impact of this as compared to say =E2=80=
=A6. Spark
> where it's all in the JVM.
>
>  Regards,
>    Bill.
>
>  --
>  Jonathan (Bill) Sparks
> Software Architecture
> Cray Inc.
>

--001a1134946c31fe48051b12df0d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div>Hi Bill,<br><br></div>Flink uses pipel=
ined data shipping by default. If you have a program like: source -&gt; map=
 -&gt; reduce -&gt; sink, the mapper will immediately start to shuffle data=
 over the network to the reducer. The reducer collects that data and starts=
 to sort it in batches. When the mapper is done and the reducer is done wit=
h the sorting, the sorted batches are merged and a sorted stream is fed int=
o the reduce function.<br></div></div>So, the shuffling and sorting happens=
 while the mapper is running, but the reduce function can only be applied a=
fter the last mapper has finished.<br><br></div>I am not aware of benchmark=
s that compare Flink&#39;s and Spark&#39;s in-memory operations, but this b=
log post compares the performance of sorting binary data in managed memory =
(like Flink) to naive sorting on the heap (Spark might do it differently!).=
<br><br>--&gt; <a href=3D"http://flink.apache.org/news/2015/05/11/Juggling-=
with-Bits-and-Bytes.html">http://flink.apache.org/news/2015/05/11/Juggling-=
with-Bits-and-Bytes.html</a><br><div><br></div><div>Fabian<br></div><div><b=
r></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">201=
5-07-17 16:07 GMT+02:00 Bill Sparks <span dir=3D"ltr">&lt;<a href=3D"mailto=
:jsparks@cray.com" target=3D"_blank">jsparks@cray.com</a>&gt;</span>:<br><b=
lockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px =
#ccc solid;padding-left:1ex">


<div style=3D"word-wrap:break-word;color:rgb(0,0,0);font-size:18px;font-fam=
ily:Calibri,sans-serif">
<div>Does flink require all the map tasks to finish before the reducers can=
 proceed like Spark, or can the reducer operations start before all the map=
pers have finished like the older Hadoop mapreduce.=C2=A0</div>
<div><br>
</div>
<div>Also my understanding is that flink manages it&#39;s own heap, do you/=
we have a sense of the performance impact of this as compared to say =E2=80=
=A6. Spark where it&#39;s all in the JVM.</div>
<div><br>
</div>
<div>Regards,</div>
<div>=C2=A0 =C2=A0Bill.</div><span class=3D"HOEnZb"><font color=3D"#888888"=
>
<div><br>
</div>
<div>
<div>--=C2=A0</div>
<div>
<div>Jonathan (Bill) Sparks</div>
<div>Software Architecture</div>
<div>Cray Inc.</div>
</div>
</div>
</font></span></div>

</blockquote></div><br></div>

--001a1134946c31fe48051b12df0d--