flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Congxian Qiu <qcx978132...@gmail.com>
Subject Re: Per Key Grained Watermark Support
Date Mon, 23 Sep 2019 11:56:33 GMT
Hi
There was a discussion about this issue[1], as the previous discussion said
at the moment this is not supported out of the box by Flink, I think you
can try keyed process function as Lasse said.

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Watermark-for-each-key-td27485.html#a27516
Best,
Congxian


Lasse Nedergaard <lassenedergaard@gmail.com> 于2019年9月23日周一 下午12:42写道:

> Hi Jiayi
>
> We have face the same challenge as we deal with IoT unit and they do not
> necessarily share the same timestamp. Watermark or. Key would be perfect
> match here. We tried to workaround with handle late events as special case
> with sideoutputs but isn’t the perfect solution.
> My conclusion is to skip watermark and create a keyed processed function
> and handle the time for each key my self.
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>
>
> Den 23. sep. 2019 kl. 06.16 skrev 廖嘉逸 <bupt_ljy@163.com>:
>
> Hi all,
>
> Currently Watermark can only be supported on task’s level(or partition
> level), which means that the data belonging to the faster key has to share
> the same watermark with the data belonging to the slower key in the same
> key group of a KeyedStream. This will lead to two problems:
>
>
> 1. Latency. For example, every key has its own window state but they have
> to trigger it after the window’s end time is exceeded by the watermark
> which is determined by the data belonging to the slowest key usually. (Same
> in CepOperator and other operators which are using watermark to fire result)
>
> 2. States Size. Because the faster key delayes its firing on result, it
> has to store more redundant states which should be pruned earlier.
>
>
> However, since the watermark has been introduced for a long time and not
> been designed to be more fine-grained in the first place, I find that it’s
> very hard to solve this problem without a big change. I wonder if there is
> anyone in community having some successful experience on this or maybe
> there is a shortcut way? If not, I can try to draft a design if this is
> needed in community.
>
>
>
> Best Regards,
>
> Jiayi Liao
>
>
>
>
>

Mime
View raw message