flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sameer Wadkar <sam...@axiomine.com>
Subject Re: Processing windows in event time order
Date Thu, 21 Jul 2016 02:03:39 GMT
Hi,

If watermarks arriving from multiple sources, how long does the Event Time Trigger wait for
the slower source to send its watermarks before triggering only from the faster source? I
have seen that if one of the sources is really slow then the elements of the faster source
fires and when the elements arrive from the slower source, the same window fires again with
the new elements only. I can work around this by adding delays but does merging watermarks
require that both have arrived by the time the watermarks progress to the point where a window
can be triggered? Is applying a delay in the watermark the only way to solve this. 

Sameer

Sent from my iPhone

> On Jul 20, 2016, at 9:41 PM, Vishnu Viswanath <vishnu.viswanath25@gmail.com> wrote:
> 
> Hi David,
> 
> You are right, the events in the window are not sorted according to the EventTime hence
the processing is not done in an increasing order of timestamp.
> As you said, you will have to do the sorting yourself in your window function to make
sure that you are processing the events in order.
> 
> What Flink does is (when EventTime is set and timestamp is assigned), it will assign
the elements to the Windows based on the EventTime, which otherwise (if using ProcessingTime)
might have ended up in a different Window. (as per the ProcessingTime).
> 
> This is as per my limited knowledge, other Flink experts can correct me if this is wrong.
> 
> Thanks,
> Vishnu 
> 
>> On Wed, Jul 20, 2016 at 9:30 PM, David Desberg <david.desberg@uber.com> wrote:
>> Hi all,
>> 
>> In Flink, after setting the time characteristic to event time and properly assigning
timestamps/watermarks, time-based windows will be created based upon event time. If we need
to process events within a window in event time order, we can sort the windowed values and
process as necessary by applying a WindowFunction. However, as I understand it, there is no
guarantee that time-based windows will be processed in time order. Is this correct? Or, if
we assume a watermarking system that (for example's sake) does not allow any late events,
is there a way within Flink to guarantee that windows will be processed (via an applied WindowFunction)
in strictly increasing time order?
>> 
>> If necessary, I can provide a more concrete explanation of what I mean/am looking
for.
>> 
>> Thanks!
>> David
> 
> 

Mime
View raw message