flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sathi Chowdhury <Sathi.Chowdh...@elliemae.com>
Subject broadcasting a stream from a collection that is populated from an web service call
Date Thu, 02 Feb 2017 21:27:24 GMT
It’s good to be here. I have a data stream coming from kinesis. I also have a list of hashmap
which holds metadata that needs to participate in the processing.
In my flink processor class I construct this metadata (hardcoded)


public static void main(String[] args) throws Exception {

…….//

DataStream<ObjectNode> jsonStream = see.addSource(new FlinkKinesisConsumer<>("int-harvester-kinesis-us-west-2",
new MyJsonDeserializationSchema(), consumerConfig));


DataStream<HashMap<String,String>> metaStream = env.fromCollection(metadata);

DataStream<String> outStream = metaStream
                                .broadcast()
                                .connect(jsonStream).flatMap(new SplitterCoFlatMapFunction());



SplitterCoFlatMapFunction extends RichCoFlatMapFunction


By doing the above I can operate on my jsonStream and avail elements of hashMap in that processing.

I am not sure , as a next step , if I am given a rest api ,which will eventually give me the
latest version of that metadata collection , what is the right way to do that call and from
where , as when flink process is started the metadata is already prepared and propagated to
the workers,where can I refresh it.


What is the most correct way of doing it?

Thanks

Sathi

=============Notice to Recipient: This e-mail transmission, and any documents, files or previous
e-mail messages attached to it may contain information that is confidential or legally privileged,
and intended for the use of the individual or entity named above. If you are not the intended
recipient, or a person responsible for delivering it to the intended recipient, you are hereby
notified that you must not read this transmission and that any disclosure, copying, printing,
distribution or use of any of the information contained in or attached to this transmission
is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately
notify the sender by telephone or return e-mail and delete the original transmission and its
attachments without reading or saving in any manner. Thank you. =============
Mime
View raw message