flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srikanth flink <flink.d...@gmail.com>
Subject Approach to match join streams to create unique streams.
Date Mon, 23 Sep 2019 11:04:39 GMT
 Hi there,

I've two streams source Kafka. Stream1 is a continuous data and stream2 is
a periodic update. Stream2 contains only one column.

*Use case*: Every entry from stream1 should verify if the stream2 has any
match.
The matched and unmatched records should be separated into new unique
streams. For example: column1, column10 from stream1 match/unmatch check on
stream2 column to put to a new stream safeStream and unSafeStream
respectively.

*Implemented solution*: stream2 as temporal function to join over stream1
which is a dynamic table.

   - Ran a time based query where stream1.column1 = stream2.column and
   stream1.column10 = stream2.column ; Working


   - Ran a time based query where stream1.column1 <> stream1.column and
   tream1.column10 <> stream1.column ; Not working.

Would like to ask if there's a possibility that I could load the stream as
a list so I could do a *contains*? OR any other approach?

Help appreciated.

Thanks
Srikanth

Mime
View raw message