flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <denis.doll...@thomsonreuters.com>
Subject RE: Equivalent of Rx combineLatest() on a join?
Date Mon, 05 Dec 2016 17:28:46 GMT
Actually that doesn't work as expected because emitted values are not purged. I'll experiment
with purging triggers and/or evictors, though I have the feeling that Flink was not designed
for what we need to do here -- but I'll keep on searching.

In the meantime any advice is appreciated. If the goal is not clear I can provide more details.

Thank you,


From: Dollfus, Denis (TR Technology & Ops)
Sent: lundi 5 d├ęcembre 2016 16:31
To: user@flink.apache.org
Subject: RE: Equivalent of Rx combineLatest() on a join?

Asking the response helped me to find the answer (yes, rubber duck debugging<https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Rubber-5Fduck-5Fdebugging&d=CwMFAw&c=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q&r=Jtkfol_Mg_eQUL17wcicdBE1Et94D7zgzS-8_bKvGlc&m=NNcjpWPJ7g8qY8aDVqKnFusNJenvEDh4gSVH9oN596Y&s=jzjcaGPpNnYAs8WTsdAtUBisrQxT234_JP59ZzJgyCw&e=>)
as it seems that the code below does what I need:

s3 = s1.join(s2)
.where(new KeySelector1()).equalTo(new KeySelector2())
                .apply(new JoinFunction<a,b,c>);

If that's a common use case (in my view it is), a syntax shortcut could help developers, e.g.
something like:

s3 = s1.join(s2)
.where(new KeySelector1()).equalTo(new KeySelector2())
                                .combineLatest(new JoinFunction<a,b,c>);


From: Dollfus, Denis (TR Technology & Ops)
Sent: lundi 5 d├ęcembre 2016 12:27
To: user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Equivalent of Rx combineLatest() on a join?

Hi all,

[first email here, I'm new to Flink, Java and Scala, sorry if I missed something obvious]

I'm exploring Flink in the context of streaming calculators. Basically, the data flow boils
down to multiple data streams with variable update rates (ms, seconds, ..., month) which are
joined before being fed to calculators. The kind of operation I need is very similar to the
Rx combineLatest<https://urldefense.proofpoint.com/v2/url?u=http-3A__reactivex.io_documentation_operators_combinelatest.html&d=CwMFAg&c=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q&r=Jtkfol_Mg_eQUL17wcicdBE1Et94D7zgzS-8_bKvGlc&m=XIk_LOnCw3IK_rNsQ5k4Y7nX14-RXxpT2wgnexTL2nM&s=HqeoR2u4r5RCZ6OjcCRP6usy9b8iUuZU3xxU6Sk17yg&e=>
operator, which results in a object being emitted whenever one of the streams is updated.

As there is no such operator predefined, I think I have to use a GlobalWindow and provide
a custom WindowAssigner. The end result would look like this (pseudo java 8 code, I hope it's

DataStream<price1> s1 = env.addSource(..);
DataStream<price2> s2 = env.addSource(..);

S3 = s1.join(s2)
.where(s1 -> id)
.equalTo(s2 -> id)
                .window(new MyCustomCombineLatestAssigner())
                .apply( ... return new object combining data from s1 and from s2);

Is the approach correct, or is there a simpler way to achieve the same join + apply mechanism

Thank you,



This e-mail is for the sole use of the intended recipient and contains information that may
be privileged and/or confidential. If you are not an intended recipient, please notify the
sender by return e-mail and delete this e-mail and any attachments. Certain required legal
entity disclosures can be accessed on our website.<http://site.thomsonreuters.com/site/disclosures/>

View raw message