flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Maintaining watermarks per key, instead of per operator instance
Date Thu, 10 Nov 2016 23:39:02 GMT
Hi Stephan,

I just wrote an answer to your SO question.

Best, Fabian

2016-11-10 11:01 GMT+01:00 Stephan Epping <stephan.epping@zweitag.de>:

> Hello,
>
> I found this question in the Nabble archive (http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Maintaining-
> watermarks-per-key-instead-of-per-operator-instance-tp7288.html) but was
> unable/dont know how to reply.
>
> Here is my question regarding the mentioned thread:
>
> Hello,
>
> I have similar requirements (see StackOverflor http://
> stackoverflow.com/questions/40465335/apache-flink-
> multiple-window-aggregations-and-late-data). I am pretty new to flink,
> could you elaborate on a possible solution? We can guarantee good ordering
> by sensor_id, thus watermarking by key would be the only reasonable way for
> us (*sensorData.keyBy('id').timeWindow(1.minute).sum('value')*), could I
> do my own watermarking after
> *sensorData.keyBy('id').overwriteWatermarking()*... per key? Or maybe
> using custom state plus a custom trigger? What happens if a sensor dies or
> is being removed completely, how can this be detected as watermarks would
> be ignored for window garbage collection. Or could we dynamically schedule
> a job of each sensor? Which would result in 1000 Jobs.
>
>
> Thanks,
> Stephan
>
>
>

Mime
View raw message