flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henrik Feldt" <henrik.fe...@voiapp.io>
Subject Re: Flink session window not progressing
Date Mon, 29 Apr 2019 17:28:51 GMT
Thinking more about this; it might just be me who is reacting to the 
sink having a zero rate of output. In fact, I have about two gigs of 
messages left in the queue until it's up to date, so I may just be 
running a slow calculation (because I've run a batch job to backfill to 
after stream). Perhaps something is broken about sink output counts?

On 29 Apr 2019, at 19:26, Henrik Feldt wrote:

> Hi guys,
>
> I'm going a PoC with Flink and I was wondering if you could help me.
>
> I've asked a question here 
> https://stackoverflow.com/questions/55907954/flink-session-window-sink-timestamp-not-progressing

> with some images. However, in summary my question is this; why doesn't 
> my session window progress?
>
> It works great when I run it against historical data, but when I run 
> it against a streaming data source (pub/sub) it sometimes gets stuck. 
> In this case, it got stuck at exactly 12:00 UTC.
>
> My window is a session window, but one where I bump the last 'edge' of 
> the window by different amounts depending on what event type it is. 
> Because some events never have other events after them.
>
> You can see the problem the easiest in these graphs, specifically the 
> one that stops at 14:00 CEST (12:00 UTC) - with green bars.
>
> ![](cid:5D5F9BCB-A282-4161-8DE3-2A711A281E63@voiapp.io "Screenshot 
> 2019-04-29 at 19.07.09.png")
>
> This graph shows the low-watermark progressing throughout the node in 
> the middle (which is also a sink); and this holds for all the nodes in 
> the graph. However, the session windowing doesn't progress, despite 
> the low-watermark progressing.
>
> ![](cid:0C3554DD-0A6E-4F2D-B101-C7F6AD51F0E9@voiapp.io "Screenshot 
> 2019-04-29 at 19.00.42.png")
>
> Regards,
> Henrik

Mime
View raw message