flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tony Wei <tony19920...@gmail.com>
Subject How to flush all window states after Kafka (0.10.x) topic was removed
Date Tue, 05 Sep 2017 06:50:41 GMT

I have a simple streaming job consuming data from Kafka and use time window
to aggregate them.
I am wondering if there is a built-in function to send a max watermark when
consumer find this topic is not available, so that the window function can
flush all state to the sink function.

My Kafka version is 0.10.x. Currently, the workaround to me is using
`TimestampAssigner` to check a specific record as termination message, and
make the watermark be Long.MAX_VALUE.
I will send this message to all partitions before I remove that topic.

I would appreciate if anyone has some suggestions. Thank you.

Best Regards,
Tony Wei

View raw message