flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Re: Multiple Async IO
Date Tue, 03 Apr 2018 17:39:11 GMT
Hi Maxim,

If reducing latency is the goal, then option #1 seems better.

Though you’d need additional logic inside of your AsyncFunction to run all 20 queries in
parallel.

I’d also consider a third option...

Use a FlatMapFunction to create 20 copies of the event (assuming it’s not large), with an
additional field indicating which query should be made.

Follow that with a rebalance(), and a single AsyncFunction that makes the appropriate query
for the event, based on this new field.

Then make sure you’ve got sufficient parallelism for your AsyncFunction to handle this fan-out.

This should let you run the queries for a single event in parallel.

— Ken


> On Apr 3, 2018, at 9:59 AM, Maxim Parkachov <lazy.gopher@gmail.com> wrote:
> 
> Hi everyone,
> 
> I'm writing streaming job which needs to query Cassandra for each event multiple times,
around 20. I would like to use Async IO for that but not sure which option to choose:
> 
> 1. Implement One AsyncFunction with 20 queries inside
> 2. Implement 20 AsyncFunctions, each with 1 query inside
> 
> Taking into account that each event needs all queries. Reduce amount of queries for each
record is not an option. 
> 
> In this case I would like to minimise processing time of event, even if throughput will
suffer. Any advice or consideration is greatly appreciated.
> 
> Thanks,
> Maxim.
>  

--------------------------------------------
http://about.me/kkrugler
+1 530-210-6378


Mime
View raw message