flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Current alternatives for async I/O
Date Sun, 09 Oct 2016 01:31:21 GMT
Hi all,

I’ve been watching the FLIP-12 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673>
design discussion, and it looks like a promising solution for the issues we’ve got with
needing to make asynchronous multi-threaded requests in a Flink operator.

What’s the best workaround with current releases of Flink?

One option is to have a special tickler source that broadcasts a Tuple0 every X milliseconds,
which gets connected to the real stream that feeds a CoFlatMap. Inside of this I’ve got
queues for incoming and generated tuples, with a thread pool to pull from the incoming and
write to the generated queues. When I get one of the “tickle” Tuple0s, I emit all of the
generated tuples.

There are issues with needing to bound the size of the queues, and all of the usual fun with
thread pools, but it seems to work. 

Is there a better/simpler approach?


— Ken

Ken Krugler
+1 530-210-6378
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

View raw message