flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: multi tenant workflow execution
Date Tue, 24 Jan 2017 10:26:21 GMT
Hi Chen,

if you plan to implement your application on top of the upcoming Flink
1.2.0 release, you might find the new AsyncFunction [1] and the
ProcessFunction [2] helpful.
AsyncFunction can be used for non-blocking calls to external services and
maintains the checkpointing semantics.
ProcessFunction allows to register and react to timers. This might easier
to use than a window for the 24h timeout.



2017-01-24 0:41 GMT+01:00 Chen Qin <qinnchen@gmail.com>:

> Hi there,
> I am researching running one flink job to support customized event driven
> workflow executions. The use case is to support running various workflows
> that listen to a set of kafka topics and performing various rpc checks, a
> user travel through multiple stages in a rule execution(workflow
> execution). e.g
> kafka topic : user click stream
> rpc checks:
> if user is member,
> if user has shown interest of signup
> ​workflows:
> ​
> workflow 1: user click -> if user is member do A then do B
> workflow 2: user click -> if user has shown interest of signup then do A
> otherwise wait for 60 mins and try recheck, expire in 24 hours
> The goal is as I said to run workflow1 & workflow2 in one flink job.
> Initial thinking describes below
> sources are series of kafka topics, all events go through coMap,cache
> lookup event -> rules mapping and fan out to multiple {rule, user} tuple.
> Based on rule definition and stage user is in a given rule, it do series of
> async rpc check and side outputs to various of sinks.
>    - If a {rule, user} tuple needs to stay in a operator states longer (1
>    day), there should be a window following async rpc checks with customized
>    purgetrigger firing those passes and purge either pass check or expired
>    tuples.
>    - If a {rule, user} execute to a stage which waits for a kafka event,
>    it should be added to cache and hookup with coMap lookups near sources
>  Does that makes sense?
> Thanks,
> Chen

View raw message