flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin Williams <colin.williams.seat...@gmail.com>
Subject Testing / Configuring event windows with Table API and SQL
Date Fri, 10 Nov 2017 03:17:38 GMT
Hello,

I've been given some flink application code and asked to implement and
ensure that our query is updated for late arriving entries. We're currently
creating a table using a Tumbling SQL query similar to the first example in


https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table/sql.html#group-windows

We then turn the result table back into a datastream using toAppendStream,
and eventually add a derivative stream to a sink. We've configured
TimeCharacteristic to event-time processing.

>From reading the documentation I was trying to configure using
withIdleStateRetentionTime, with the expectation that this setting would
allow me to deal with late arrivals past a given watermark time, but within
the retention time.

Then to test this I created a simple source which triggers the watermark,
so that I'd have next a late arrival. However so far the watermark seems to
cause something to discriminate the late arrival. Then in my test sink
where I'm trying to capture all emitted outputs, and hopefully the updated
value I don't find one.

So it seems that my understanding of how to deal with late events, or my
test platform is wrong. Can anyone recognize what I'm doing wrong?


Best,

Colin Williams

Mime
View raw message