flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chirag Dewan <chirag.dewa...@yahoo.in>
Subject Re: Broadcast state before events stream consumption
Date Fri, 08 Feb 2019 16:48:47 GMT
 Hi Vadim,
I would be interested in this too. 
Presently, I have to read my lookup source in the open method and keep it in a cache. By doing
that I cannot make use of the broadcast state until ofcourse the first emit comes on the Broadcast
The problem with waiting the event stream is the lack of knowledge that I have read all the
data from the lookup source. There is no possibility of having a special marker in the data
as well for my use case.
So pre loading the data seems to be the only option right now.

    On Friday, 8 February, 2019, 7:45:37 pm IST, Vadim Vararu <vadim.vararu@adswizz.com>
  Hi all,
I need to use the broadcast state mechanism (https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html)
for the next scenario.
I have a reference data stream (slow) and an events stream (fast running) and I want to do
a kind of lookup in the reference stream for eachevent. The broadcast state mechanism seems
to fit perfect the scenario. 
>From documentation:As an example where broadcast state can emerge as a natural fit, one
can imagine a low-throughput stream containing a set of rules which we want to evaluate against
all elements coming from another stream.

However, I am not sure what is the correct way to delay the consumption of the fast running
stream until the slow one is fully read (in case of a file) or until a marker is emitted (in
case of some other source). Is there any way to accomplish that? It doesn't seem to be a rare
use case.
Thanks, Vadim.  
View raw message