flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: A use-case for Flink and reactive systems
Date Wed, 04 Jul 2018 21:16:53 GMT
Looks interesting.

As I understand you have a microservice based on ingestion where a topic is
defined for streaming messages that include transactional data. These
transactions should already exist in your DB. For now we look at DB as part
of your microservices and we take a logical view of it.

So


   1. First microservice M1 provides ingestion of kafka yopic
   2. Second microservice M2 deploys Flink or Spark Streaming to manipulate
   the incoming messages. We can look at this later.
   3. Third microservice M3 consist of the database that provides current
   records for accounts identified by the account number in your message queue
   4. M3 will have to validate the incoming account number, update the
   transaction and provide completion handshake. Effectively you are providing
   DPaaS


So far we have avoided interfaces among these services. But I gather M1 and
M2 are well established. Assuming that Couchbase is your choice of DB I
believe it provides JDBC drivers of some sort. It does not have to be
Couchbase. You can achieve the same with Hbase as well or MongoDB. Anyhow
the only challenge I see here is the interface between your Flink
application in M2 and M3

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 4 Jul 2018 at 17:56, Yersinia Ruckeri <yersiniaruckeri@gmail.com>
wrote:

> Hi all,
>
> I am working on a prototype which should include Flink in a reactive
> systems software. The easiest use-case with a traditional bank system where
> I have one microservice for transactions and another one for
> account/balances.
> Both are connected with Kafka.
>
> Transactions record a transaction and then send, via Kafka, a message
> which include account identifer and the amount.
> On the other microservice I want to have Flink consuming this topic and
> updating the balance of my account based on the incoming message. it needs
> to pull from the DB my data and make the update.
> The DB is Couchbase.
>
> I spent few hours online today, but so far I only found there's no sink
> for Couchbase, I need to build one which shouldn't be a big deal. I haven't
> found information on how I can make Flink able to interact with a DB to
> retrieve information and store information.
>
> I guess the case is a good case, as updating state in an event sourcing
> based application is part of the first page of the manual. I am not looking
> to just dump a state into a DB, but to interact with the DB: retrieving
> data, elaborating them with the input coming from my queue, and persisting
> them (especially if I want to make a second prototype using Flink CEP).
>
> I probably even read how to do it, but I didn't recognize.
>
> Can anybody help me to figure out better this part?
>
> Thanks,
> Y.
>

Mime
View raw message