flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yassine MARZOUGUI <y.marzou...@mindlytix.com>
Subject Re: Windows emit results at the end of the stream
Date Thu, 23 Mar 2017 10:45:32 GMT
Hi Sonex,

When using readTextFile(...) with event time, only one watermark with the
value Long.MAX_VALUE is sent at the end of the stream, which explais why
the windows are stored until the whole file is processed. In order to have
periodic watermarks, you need to process the file continuousely as folows:

TextInputFormat inputFormat = new TextInputFormat(new Path("file/to/read.txt"));

Hope this helps.


2017-03-23 9:47 GMT+01:00 Sonex <alfredjenssen@gmail.com>:

> Hi everyone,
> I am using a simple window computation on a stream with event time. The
> code
> looks like this:
> streamData.readTextFile(...)
>     .map(...)
>     .assignAscendingTimestamps(_.timestamp)
>     .keyBy(_.id)
>     .timeWindow(Time.seconds(3600),Time.seconds(3600))
>     .apply(new MyWindowFunction)
>     .map(...)
> By monitoring the memory usage and the flink web dashboard, I noticed that
> flink applies the window function until the entire stream finishes (thus
> storing all aggregations in memory) and then continues to the map
> transformation. What I would expect is emission of window results to the
> map
> transformation as soon as results of the window are ready.
> Can anyone explain this behavior?
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Windows-emit-
> results-at-the-end-of-the-stream-tp12337.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.

View raw message