flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hequn Cheng <chenghe...@gmail.com>
Subject Re: Counting elements that appear "behind" the watermark
Date Tue, 31 Jul 2018 01:20:50 GMT
Hi Julio,

If I understand correctly, you want to adjust your watermarks automatically?
It is true that there are no direct ways to get metric from the
AssignerWithPeriodicWatermarks. Adding ProcessFunction before
assignTimestampsAndWatermarks seems a solution. In the ProcessFunction, you
can count the late number and send the number to the downstream
assignTimestampsAndWatermarks to adjust watermarks.

Best, Hequn

On Tue, Jul 31, 2018 at 1:32 AM, Julio Biason <julio.biason@azion.com>
wrote:

> Hello,
>
> Our current watermark model is "some time behind the most recent seen
> element" (very close to what the docs have in "Periodic Watermark"
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_
> timestamps_watermarks.html#with-periodic-watermarks). It fits our current
> processing model.
>
> The thing is, we want to extract information about elements appearing
> behind the watermark, to give some insight when we need to update the
> amount of time behind the most seen element we need. The problem is, I
> can't create any metrics inside the AssignerWithPeriodicWatermarks 'cause
> it has no `getRuntime()` to attach the metric.
>
> Is there any way we can count those (a ProcessFunction before the .
> assignTimestampsAndWatermarks(), maybe)?
>
> --
> *Julio Biason*, Sofware Engineer
> *AZION*  |  Deliver. Accelerate. Protect.
> Office: +55 51 3083 8101 <callto:+555130838101>  |  Mobile: +55 51
> <callto:+5551996209291>*99907 0554*
>

Mime
View raw message