flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: REST API call in stream transformation
Date Thu, 27 Apr 2017 16:17:20 GMT
Hi Vijay,

Generally, for asynchronous operations to enrich (or in your case, fetching the algorithm
for the actual transformation of the data), you’ll want to look at Flink’s Async I/O [1].

For your second question, I can see it as a stateful `FlatMapFunction` that keeps the seen
results as managed state. Once all results are seen (i.e. the last result arrives at the operator),
you join them and emit your final result further downstream.

Does this help with the use case you have in mind?


[1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/asyncio.html

On 27 April 2017 at 5:29:05 PM, G.S.Vijay Raajaa (gsvijayraajaa@gmail.com) wrote:


I have just started to explore Flink and have couple of questions. I  am wondering if its
possible to call a rest endpoint asynchronously and pipe the response to the next state of
my transformation on the stream. The idea is such that after charging my data in a predefined
time window, I would like to apply some algorithm/transformation on the window of data external
to Flink. The algos have been exposed as REST endpoints.

My seconds question is an extension to the previous one, if i need to transform my window
data by applying three different algorithms in parallel exposed via subsequent rest endpoints,
how do wait unless the individual rest endpoints respond back . Potentially I need to join
the results of the three algorithms before I call the sink.

Vijay Raajaa 
View raw message