flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philipp Bussche <philipp.buss...@gmail.com>
Subject Re: Enriching a tuple mapped from a datastream with data coming from a JDBC source
Date Tue, 04 Oct 2016 11:49:53 GMT
Awesome, thanks Fabian !

I will give this a try.


Fabian Hueske-2 wrote
> Hi Philipp,
> 
> If I got your requirements right you would like to:
> 1) load an initial hashmap via JDBC
> 2) update the hashmap from a stream
> 3) use the hashmap to enrich another stream.
> 
> You can use a CoFlatMap to do this:
> 
> stream1.connect(stream2).flatMap(new YourCoFlatMapFunction).
> 
> YourCoFlatMapFunction should implement RichCoFlatMapFunction. The initial
> hashmap load can be done in the open method.
> A CoFlatMapFunction has two inputs and two flatMap methods, one for each
> input. One method can update the hashmap, the other enrich the second
> stream with data from the hashmap.
> Both methods are not concurrently called and the order in which they are
> called depends on what data is available.
> 
> In general, it is not possible to share local operator state among
> different operators (or even parallel instance of the same operator).
> 
> Hope this helps,
> Fabian


Fabian Hueske-2 wrote
> Hi Philipp,
> 
> If I got your requirements right you would like to:
> 1) load an initial hashmap via JDBC
> 2) update the hashmap from a stream
> 3) use the hashmap to enrich another stream.
> 
> You can use a CoFlatMap to do this:
> 
> stream1.connect(stream2).flatMap(new YourCoFlatMapFunction).
> 
> YourCoFlatMapFunction should implement RichCoFlatMapFunction. The initial
> hashmap load can be done in the open method.
> A CoFlatMapFunction has two inputs and two flatMap methods, one for each
> input. One method can update the hashmap, the other enrich the second
> stream with data from the hashmap.
> Both methods are not concurrently called and the order in which they are
> called depends on what data is available.
> 
> In general, it is not possible to share local operator state among
> different operators (or even parallel instance of the same operator).
> 
> Hope this helps,
> Fabian





--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Enriching-a-tuple-mapped-from-a-datastream-with-data-coming-from-a-JDBC-source-tp8993p9313.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message