Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <CAMsP=1Sua0+z-n7RBBa+i7NvGXo8sax56_jZAvNdzUCwC-Rdsw@mail.gmail.com>
References: <CAMsP=1Sua0+z-n7RBBa+i7NvGXo8sax56_jZAvNdzUCwC-Rdsw@mail.gmail.com>
From: Fabian Hueske <fhueske@gmail.com>
Date: Fri, 18 Nov 2016 09:40:36 +0100
Message-ID: <CAAdrtT2_Ye55tMfLNtKhQ+NjE2Xr_six8vr84_4hqi+-kEDR8g@mail.gmail.com>
Subject: Re: Cross product of datastream and dataset
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1142b65260b0d105418f4429
archived-at: Fri, 18 Nov 2016 08:41:23 -0000

--001a1142b65260b0d105418f4429
Content-Type: text/plain; charset=UTF-8

Hi,

it is not possible to mix the DataSet and DataStream APIs at the moment.

If the DataSet is constant and not too big (which I assume, since otherwise
crossing would be extremely expensive), you can load the data into a
stateful MapFunction.
For that you can implement a RichFlatMapFunction and read the data in
open(). For each incoming record, i.e., each call of map(), you cross it
with the records in the state and immediately evaluate the condition and
count. That way you don't generate too many records.

If your DataSet is slowly changing, you can think of using a stateful
CoFlatmapFunction and use on input to read the stream and the other to
update the dataset.

Hope this helps,
Fabian


2016-11-17 22:36 GMT+01:00 Charlie Moad <charlie.moad@geofeedia.com>:

> We're having trouble mapping our problem to Flink.
>
> - For each incoming item
> - Generate tuples of the item crossed with a data set
> - Filter the tuples based on a condition
> - Know the count of matching tuples
>
> This seems to be mashup of DataStream and DataSet, but it appears you
> can't operate with those together. We are also wondering if kicking off a
> batch job for each incoming item is a feasible approach. We don't have time
> window constraints.
>
> Any recommendations would be greatly appreciated.
>
> - Charlie
>
>

--001a1142b65260b0d105418f4429
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div><div>Hi,<br><br></div>it is =
not possible to mix the DataSet and DataStream APIs at the moment. <br></di=
v><br></div>If the DataSet is constant and not too big (which I assume, sin=
ce otherwise crossing would be extremely expensive), you can load the data =
into a stateful MapFunction.<br></div>For that you can implement a RichFlat=
MapFunction and read the data in open(). For each incoming record, i.e., ea=
ch call of map(), you cross it with the records in the state and immediatel=
y evaluate the condition and count. That way you don&#39;t generate too man=
y records.<br><br></div>If your DataSet is slowly changing, you can think o=
f using a stateful CoFlatmapFunction and use on input to read the stream an=
d the other to update the dataset.<br><br></div>Hope this helps,<br></div>F=
abian<br><div><div><div><div><br></div></div></div></div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">2016-11-17 22:36 GMT+01:00 =
Charlie Moad <span dir=3D"ltr">&lt;<a href=3D"mailto:charlie.moad@geofeedia=
.com" target=3D"_blank">charlie.moad@geofeedia.com</a>&gt;</span>:<br><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #cc=
c solid;padding-left:1ex"><div dir=3D"ltr">We&#39;re having trouble mapping=
 our problem to Flink.<div><br></div><div>- For each incoming item</div><di=
v>- Generate tuples of the item crossed with a data set</div><div>- Filter =
the tuples based on a condition</div><div>- Know the count of matching tupl=
es</div><div><br></div><div>This seems to be mashup of DataStream and DataS=
et, but it appears you can&#39;t operate with those together. We are also w=
ondering if kicking off a batch job for each incoming item is a feasible ap=
proach. We don&#39;t have time window constraints.</div><div><br></div><div=
>Any recommendations would be greatly appreciated.</div><span class=3D"HOEn=
Zb"><font color=3D"#888888"><div><br></div><div>- Charlie</div><div><br></d=
iv><div>
</div></font></span></div>
</blockquote></div><br></div>

--001a1142b65260b0d105418f4429--