flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Size of a window without explicit trigger/evictor
Date Fri, 18 Dec 2015 10:56:59 GMT
Hi Nirmalya,

sorry for the delayed answer.

First of all, Flink does not take care that our windows fit into memory.
The default trigger depends on the way in which you define a window. Given
a KeyedStream you can define a window in the following ways:

KeyedStream s = ...
s.timeWindow() // this will use an EventTimeTrigger or
ProcessingTimeTrigger, depending on the time characteristics of the stream
s.countWindow() // this will use a CountTrigger
s.window(? extends WindowAssigner) // this will use the default trigger as
defined by the WindowAssigner

None of these triggers monitors the JVM heap to prevent OOMs. If you define
a TimeTrigger for one hour and receive too much data, the program will
crash. IMO, this behavior is preferable over early triggering which would
cause semantically wrong results. If you use a ReduceFunction to compute
the result of a window (and no Evictor), the window result can be partially
aggregated and its state does not grow.

Best, Fabian

2015-12-10 2:47 GMT+01:00 Nirmalya Sengupta <sengupta.nirmalya@gmail.com>:

> Hello Fabian <fhueske@gmail.com>
>
> A small question: during the course of our recent conversation on the
> behaviour of window,trigger and evictor, you had mentioned that if I - the
> application programmer - do not attach a trigger to a window, Flink will
> attach one by itself. This trigger ensures that the size of the window
> never grows beyond a threshold, thereby ensuring that a burgeoning window
> never inflicts a OOM on Flink.
>
> Is this a special Trigger? What's the name of the class? Moreover, how is
> that threshold size (of the window) determined? Is it configurable?
>
> TIA.
>
> -- Nirmalya
>
> --
> Software Technologist
> http://www.linkedin.com/in/nirmalyasengupta
> "If you have built castles in the air, your work need not be lost. That is
> where they should be.
> Now put the foundation under them."
>

Mime
View raw message