Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
From: Elias Levy <fearsome.lucidity@gmail.com>
Date: Thu, 11 Aug 2016 17:51:49 -0700
Message-ID: <CAFDmHdPDiENdzoZZXsyjsHsTY1Yz59gG6rdY9C-F=OY6qB+e=Q@mail.gmail.com>
Subject: Does Flink DataStreams using combiners?
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11401dcaddcc540539d54ada
archived-at: Fri, 12 Aug 2016 00:52:14 -0000

--001a11401dcaddcc540539d54ada
Content-Type: text/plain; charset=UTF-8

I am wondering if Flink makes use of combiners to pre-reduce a keyed and
windowed stream before shuffling the data among workers.

I.e. will it use a combiner in something like:

stream.flatMap {...}
      .assignTimestampsAndWatermarks(...)
      .keyBy(...)
      .timeWindow(...)
      .trigger(...)
      .sum("cnt")

or will it shuffle the keyed input before the sum reduction?

If it does make use of combiners, it would be useful to point this out in
the documentation, particularly if it only applies to certain types of
reducers, folds, etc.

--001a11401dcaddcc540539d54ada
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I am wondering if Flink makes use of combiners to pre-redu=
ce a keyed and windowed stream before shuffling the data among workers.<div=
><br></div><div>I.e. will it use a combiner in something like:</div><div><b=
r></div><div><div>stream.flatMap {...}</div><div>=C2=A0 =C2=A0 =C2=A0 .assi=
gnTimestampsAndWatermarks(...)</div><div>=C2=A0 =C2=A0 =C2=A0 .keyBy(...)<b=
r></div><div>=C2=A0 =C2=A0 =C2=A0 .timeWindow(...)</div><div>=C2=A0 =C2=A0 =
=C2=A0 .trigger(...)<br></div><div>=C2=A0 =C2=A0 =C2=A0 .sum(&quot;cnt&quot=
;)<br></div><div><br></div></div><div>or will it shuffle the keyed input be=
fore the sum reduction?</div><div><br></div><div>If it does make use of com=
biners, it would be useful to point this out in the documentation, particul=
arly if it only applies to certain types of reducers, folds, etc.</div></di=
v>

--001a11401dcaddcc540539d54ada--