flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yury Ruchin <yuri.ruc...@gmail.com>
Subject Blocking RichFunction.open() and backpressure
Date Sat, 17 Dec 2016 12:42:42 GMT
Hi all,

I have a streaming job that essentially looks like this: KafkaSource -> Map
-> EventTimeWindow -> RichFlatMap -> CustomSink. The RichFlatMap part does
some heavy lifting in open(), so that the open() call blocks for several
minutes. I assumed that until open() returns the backpressure mechanism
would slow down the entire upstream up to the KafkaSource, so that no new
records would be emitted to the pipeline until the RichFlatMap is ready.
What I actually observe is that the KafkaSource, Map and EventTimeWindow
continue processing - the in/out records, in/out MB counters keep
increasing. The RichFlatMap and its downstream CustomSink have 0 as
expected, until the RichFlatMap is actually done with open(). The
backpressure monitor in Flink UI shows "OK" for all operators.

Why doesn't backpressure mechanism work in this case?

Thanks,
Yury

Mime
View raw message