Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAEr=Az19c8cdUfBsPbFpCPXAn_gSNzb9DkCZ1qPHWUKJKWRqYQ@mail.gmail.com>
References: 
 <CAEr=Az19c8cdUfBsPbFpCPXAn_gSNzb9DkCZ1qPHWUKJKWRqYQ@mail.gmail.com>
From: Fabian Hueske <fhueske@gmail.com>
Date: Fri, 18 Dec 2015 11:56:59 +0100
Message-ID: 
 <CAAdrtT3go0Sd4BkMQ+S3YBNftV=ASojbvMUrHanc8NGEWxkrog@mail.gmail.com>
Subject: Re: Size of a window without explicit trigger/evictor
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a114020ac63401c05272a0107

--001a114020ac63401c05272a0107
Content-Type: text/plain; charset=UTF-8

Hi Nirmalya,

sorry for the delayed answer.

First of all, Flink does not take care that our windows fit into memory.
The default trigger depends on the way in which you define a window. Given
a KeyedStream you can define a window in the following ways:

KeyedStream s = ...
s.timeWindow() // this will use an EventTimeTrigger or
ProcessingTimeTrigger, depending on the time characteristics of the stream
s.countWindow() // this will use a CountTrigger
s.window(? extends WindowAssigner) // this will use the default trigger as
defined by the WindowAssigner

None of these triggers monitors the JVM heap to prevent OOMs. If you define
a TimeTrigger for one hour and receive too much data, the program will
crash. IMO, this behavior is preferable over early triggering which would
cause semantically wrong results. If you use a ReduceFunction to compute
the result of a window (and no Evictor), the window result can be partially
aggregated and its state does not grow.

Best, Fabian

2015-12-10 2:47 GMT+01:00 Nirmalya Sengupta <sengupta.nirmalya@gmail.com>:

> Hello Fabian <fhueske@gmail.com>
>
> A small question: during the course of our recent conversation on the
> behaviour of window,trigger and evictor, you had mentioned that if I - the
> application programmer - do not attach a trigger to a window, Flink will
> attach one by itself. This trigger ensures that the size of the window
> never grows beyond a threshold, thereby ensuring that a burgeoning window
> never inflicts a OOM on Flink.
>
> Is this a special Trigger? What's the name of the class? Moreover, how is
> that threshold size (of the window) determined? Is it configurable?
>
> TIA.
>
> -- Nirmalya
>
> --
> Software Technologist
> http://www.linkedin.com/in/nirmalyasengupta
> "If you have built castles in the air, your work need not be lost. That is
> where they should be.
> Now put the foundation under them."
>

--001a114020ac63401c05272a0107
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div>Hi Nirmalya,<br><br></div>sorry for the del=
ayed answer.<br><br></div>First of all, Flink does not take care that our w=
indows fit into memory. <br>The default trigger depends on the way in which=
 you define a window. Given a KeyedStream you can define a window in the fo=
llowing ways:<br><br></div><div>KeyedStream s =3D ...<br></div><div>s.timeW=
indow() // this will use an EventTimeTrigger or ProcessingTimeTrigger, depe=
nding on the time characteristics of the stream<br></div><div>s.countWindow=
() // this will use a CountTrigger<br></div><div>s.window(? extends WindowA=
ssigner) // this will use the default trigger as defined by the WindowAssig=
ner<br><br></div><div>None of these triggers monitors the JVM heap to preve=
nt OOMs. If you define a TimeTrigger for one hour and receive too much data=
, the program will crash. IMO, this behavior is preferable over early trigg=
ering which would cause semantically wrong results. If you use a ReduceFunc=
tion to compute the result of a window (and no Evictor), the window result =
can be partially aggregated and its state does not grow.<br><br></div><div>=
Best, Fabian<br></div></div><div class=3D"gmail_extra"><br><div class=3D"gm=
ail_quote">2015-12-10 2:47 GMT+01:00 Nirmalya Sengupta <span dir=3D"ltr">&l=
t;<a href=3D"mailto:sengupta.nirmalya@gmail.com" target=3D"_blank">sengupta=
.nirmalya@gmail.com</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
dir=3D"ltr">Hello Fabian &lt;<a href=3D"mailto:fhueske@gmail.com" target=3D=
"_blank">fhueske@gmail.com</a>&gt;<div><br></div><div>A small question: dur=
ing the course of our recent conversation on the behaviour of window,trigge=
r and evictor, you had mentioned that if I - the application programmer - d=
o not attach a trigger to a window, Flink will attach one by itself. This t=
rigger ensures that the size of the window never grows beyond a threshold, =
thereby ensuring that a burgeoning window never inflicts a OOM on Flink.</d=
iv><div><br></div><div>Is this a special Trigger? What&#39;s the name of th=
e class? Moreover, how is that threshold size (of the window) determined? I=
s it configurable?</div><div><br></div><div>TIA.</div><span class=3D"HOEnZb=
"><font color=3D"#888888"><div><br></div><div>-- Nirmalya<br clear=3D"all">=
<div><br></div>-- <br><div>Software Technologist<br><a href=3D"http://www.l=
inkedin.com/in/nirmalyasengupta" target=3D"_blank">http://www.linkedin.com/=
in/nirmalyasengupta</a><br>&quot;If you have built castles in the air, your=
 work need not be lost. That is where they should be.<br>Now put the founda=
tion under them.&quot;</div>
</div></font></span></div>
</blockquote></div><br></div>

--001a114020ac63401c05272a0107--