Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <2A57CC9D-A649-47D0-8DCE-BF965D62A8FF@uber.com>
References: <2A57CC9D-A649-47D0-8DCE-BF965D62A8FF@uber.com>
From: Vishnu Viswanath <vishnu.viswanath25@gmail.com>
Date: Wed, 20 Jul 2016 21:41:58 -0400
Message-ID: <CAK2z7j4=vjAibkR0O6tAkt9ew0ik=bWRXObHsRqZ1qZiuBFZ4g@mail.gmail.com>
Subject: Re: Processing windows in event time order
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11457c06a10fec05381b6ddc
archived-at: Thu, 21 Jul 2016 01:42:29 -0000

--001a11457c06a10fec05381b6ddc
Content-Type: text/plain; charset=UTF-8

Hi David,

You are right, the events in the window are not sorted according to the
EventTime hence the processing is not done in an increasing order of
timestamp.
As you said, you will have to do the sorting yourself in your window
function to make sure that you are processing the events in order.

What Flink does is (when EventTime is set and timestamp is assigned), it
will assign the elements to the Windows based on the EventTime, which
otherwise (if using ProcessingTime) might have ended up in a different
Window. (as per the ProcessingTime).

This is as per my limited knowledge, other Flink experts can correct me if
this is wrong.

Thanks,
Vishnu

On Wed, Jul 20, 2016 at 9:30 PM, David Desberg <david.desberg@uber.com>
wrote:

> Hi all,
>
> In Flink, after setting the time characteristic to event time and properly
> assigning timestamps/watermarks, time-based windows will be created based
> upon event time. If we need to process events within a window in event time
> order, we can sort the windowed values and process as necessary by applying
> a WindowFunction. However, as I understand it, there is no guarantee that
> time-based windows will be processed in time order. Is this correct? Or, if
> we assume a watermarking system that (for example's sake) does not allow
> any late events, is there a way within Flink to guarantee that windows will
> be processed (via an applied WindowFunction) in strictly increasing time
> order?
>
> If necessary, I can provide a more concrete explanation of what I mean/am
> looking for.
>
> Thanks!
> David

--001a11457c06a10fec05381b6ddc
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi David,</div><div><br></div><div>You are right, the=
 events in the window are not sorted according to the EventTime hence the p=
rocessing is not done in an increasing order of timestamp.</div><div>As you=
 said, you will have to do the sorting yourself in your window function to =
make sure that you are processing the events in order.</div><div><br></div>=
<div>What Flink does is (when EventTime is set and timestamp is assigned), =
it will assign the elements to the Windows based on the EventTime, which ot=
herwise (if using ProcessingTime) might have ended up in a different Window=
. (as per the ProcessingTime).</div><div><br></div><div>This is as per my l=
imited knowledge, other Flink experts can correct me if this is wrong.</div=
><div><br></div><div>Thanks,</div><div>Vishnu=C2=A0</div><div class=3D"gmai=
l_extra"><br><div class=3D"gmail_quote">On Wed, Jul 20, 2016 at 9:30 PM, Da=
vid Desberg <span dir=3D"ltr">&lt;<a href=3D"mailto:david.desberg@uber.com"=
 target=3D"_blank">david.desberg@uber.com</a>&gt;</span> wrote:<br><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc s=
olid;padding-left:1ex">Hi all,<br>
<br>
In Flink, after setting the time characteristic to event time and properly =
assigning timestamps/watermarks, time-based windows will be created based u=
pon event time. If we need to process events within a window in event time =
order, we can sort the windowed values and process as necessary by applying=
 a WindowFunction. However, as I understand it, there is no guarantee that =
time-based windows will be processed in time order. Is this correct? Or, if=
 we assume a watermarking system that (for example&#39;s sake) does not all=
ow any late events, is there a way within Flink to guarantee that windows w=
ill be processed (via an applied WindowFunction) in strictly increasing tim=
e order?<br>
<br>
If necessary, I can provide a more concrete explanation of what I mean/am l=
ooking for.<br>
<br>
Thanks!<br>
<span class=3D"HOEnZb"><font color=3D"#888888">David</font></span></blockqu=
ote></div><div><br></div>
</div></div>

--001a11457c06a10fec05381b6ddc--