Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
From: Sameer W <sameer@axiomine.com>
Date: Mon, 3 Oct 2016 08:31:06 -0400
Message-ID: <CANvT7L1qRdHztVfbixhApMqLgfVd8-Ko1AQN5ASFq6AgvWnFBQ@mail.gmail.com>
Subject: Side Inputs vs. Connected Streams
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=f403045dd56032d360053df51e8d
archived-at: Mon, 03 Oct 2016 12:31:27 -0000

--f403045dd56032d360053df51e8d
Content-Type: text/plain; charset=UTF-8

Hi,

I read the Side Inputs
<https://docs.google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit#>
design document. How does it compare to using ConnectedStreams with respect
to handling the ordering of streams transparently?

One of the challenges I have with ConnectedStreams is I need to buffer main
input if the rules stream has not arrived yet. Does this automatically go
away with Side Inputs? Will the call to  String sideValue =

       getRuntimeContext().getSideInput(filterString);
block if the side input is not available yet? And is the reverse also true?

Alternatively, if my rules are not large in number and I want to broadcast
them to all nodes is the below equivalent to using SideInputs where side
inputs are broadcast to all nodes and ensure that the side input is
evaluated before the main input:

DataStream ds4 = ds3.connect(dsSide.broadcast());

Will the above ensure that dsSide is always available before ds3 elements
arrive on the connected stream. Am I correct in assuming that ds2 changes
will continue to be broadcast to ds3 (with no ordering guarantees between
ds3 and dsSide, ofcourse).


Thanks,
Sameer

--f403045dd56032d360053df51e8d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>I read the <a href=3D"https://docs.=
google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit#">S=
ide Inputs</a> design document. How does it compare to using ConnectedStrea=
ms with respect to handling the ordering of streams transparently?=C2=A0</d=
iv><div><br></div><div>One of the challenges I have with ConnectedStreams i=
s I need to buffer main input if the rules stream has not arrived yet. Does=
 this automatically go away with Side Inputs? Will the call to=C2=A0<span s=
tyle=3D"background-color:transparent;color:rgb(0,0,0);font-family:&quot;cou=
rier new&quot;;font-size:14.6667px;white-space:pre-wrap"> </span><span styl=
e=3D"background-color:transparent;color:rgb(0,0,0);white-space:pre-wrap"><f=
ont face=3D"monospace, monospace">String sideValue =3D</font></span></div><=
span id=3D"gmail-docs-internal-guid-c714ee7b-8a77-7898-ef08-6ed672c8138d"><=
p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span style=3D"color:rgb(0,0,0);background-color:transparent;vertical-align:=
baseline;white-space:pre-wrap"><font face=3D"monospace, monospace"> =C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0getRuntimeContext().getSideInput(filter=
String);</font></span></p><div>block if the side input is not available yet=
? And is the reverse also true?<br></div><div><br></div><div>Alternatively,=
 if my rules are not large in number and I want to broadcast them to all no=
des is the below equivalent to using SideInputs where side inputs are broad=
cast to all nodes and ensure that the side input is evaluated before the ma=
in input:</div><div><br></div><div><font face=3D"monospace, monospace">Data=
Stream ds4 =3D ds3.connect(</font><span style=3D"font-family:monospace,mono=
space">dsSide.broadcast()</span><font face=3D"monospace, monospace">);</fon=
t><br></div><div><font face=3D"monospace, monospace"><br></font></div><div>=
Will the above ensure that dsSide is always available before ds3 elements a=
rrive on the connected stream. Am I correct in assuming that ds2 changes wi=
ll continue to be broadcast to ds3 (with no ordering guarantees between ds3=
 and dsSide, ofcourse).<font face=3D"monospace, monospace"><br></font></div=
><div><br></div><div><br></div><div>Thanks,</div><div>Sameer</div><div><br>=
</div><div><br></div><div><br></div></span></div>

--f403045dd56032d360053df51e8d--