flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Federico D'Ambrosio" <federico.dambro...@smartlab.ws>
Subject How to deal with blocking calls inside a Sink?
Date Mon, 02 Oct 2017 09:53:27 GMT
Hi, I've implemented a sink for Hive as a RichSinkFunction, but once I've
integrated it in my current flink job, I noticed that the processing of the
events slowed down really bad, I guess because of some blocking calls that
need to be when interacting with hive streaming api.

So, what can be done to make it so the throughput doesn't get hurt by these
calls? I guess increasing (by a lot) the parallelism of the sink operator
could be a solution, but I'd think it's not really a good one.

Maybe using the AsyncFunction API? Decoupling the sink in a buffer which
sends the data + operations to be made in the asyncInvoke method of the
AsyncFunction?

Any suggestion is appreciated.
Kind regards,
Federico D'Ambrosio

Mime
View raw message