flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Al-Isawi Rami <Rami.Al-Is...@comptel.com>
Subject Re: Combining streams with static data and using REST API as a sink
Date Mon, 23 May 2016 13:35:17 GMT
Hi Josh,

I am no expert in Flink yet, but here are my thoughts on this:

1. what about you stream an event to flink everytime the DB of items have an update? then
in some background thread you get the new data from the DB let it be through REST (if it is
only few updates a day) then load the results in memory and there is your updated static data.

2. REST API are over HTTP, how that is possible to be a sink? does not sound like flink job
at all to serve http requests. simply sink the results to some DB and have some component
to read from DB and serve it as REST API.

-Rami

On 23 May 2016, at 16:22, Josh <jofo90@gmail.com<mailto:jofo90@gmail.com>> wrote:


Hi all,

I am new to Flink and have a couple of questions which I've had trouble finding answers to
online. Any advice would be much appreciated!

  1.  What's a typical way of handling the scenario where you want to join streaming data
with a (relatively) static data source? For example, if I have a stream 'orders' where each
order has an 'item_id', and I want to join this stream with my database of 'items'. The database
of items is mostly static (with perhaps a few new items added every day). The database can
be retrieved either directly from a standard SQL database (postgres) or via a REST call. I
guess one way to handle this would be to distribute the database of items with the Flink tasks,
and to redeploy the entire job if the items database changes. But I think there's probably
a better way to do it?
  2.  I'd like my Flink job to output state to a REST API. (i.e. using the REST API as a sink).
Updates would be incremental, e.g. the job would output tumbling window counts which need
to be added to some property on a REST resource, so I'd probably implement this as a PATCH.
I haven't found much evidence that anyone else has used a REST API as a Flink sink - is there
a reason why this might be a bad idea?

Thanks for any advice on these,

Josh

Disclaimer: This message and any attachments thereto are intended solely for the addressed
recipient(s) and may contain confidential information. If you are not the intended recipient,
please notify the sender by reply e-mail and delete the e-mail (including any attachments
thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination
or other use of, or taking of any action in reliance upon, this information by persons or
entities other than the intended recipient(s) is prohibited. Thank you.

Mime
View raw message