flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <k.klou...@data-artisans.com>
Subject Re: Regarding Late Elements
Date Tue, 04 Oct 2016 15:04:52 GMT
Hi Vinay,

From what I understand from your code, the only difference of your trigger compared to the

one shipping with Flink is that for the late elements, instead of firing and keeping the element,

you fire and purge, i.e. clean the window state.

This does not solve the problem of dropping the super late elements if their window has expired
(currentWatermark >= window.maxTimestamp + allowedLateness).

Cheers,
Kostas

> On Oct 4, 2016, at 3:25 PM, vinay patil <vinay18.patil@gmail.com> wrote:
> 
> Hi Kostas,
> 
> Thank you for your reply, yes that will be a good functionality to have, but for now
the Custom Trigger as close to 1.0.3 works for me.
> 
> 
> public TriggerResult onElement(Object element, long timestamp, TimeWindow window, TriggerContext
ctx) throws Exception {
> 		if(window.maxTimestamp() <= ctx.getCurrentWatermark()) {
>              return TriggerResult.FIRE_AND_PURGE;
>         } else {
>                ctx.registerEventTimeTimer(window.maxTimestamp());
> 		       return TriggerResult.CONTINUE;
>         }
> 	}
> public TriggerResult onEventTime(long time, TimeWindow window, TriggerContext ctx) {
return TriggerResult.FIRE_AND_PURGE; }
> 
> This is the change I have done in my custom trigger which is similar to 1.0.3 , does
this looks good to you ?
> (I am not using canMerge and onMerge methods since I am not aware of it)
> 
> 
> Regards,
> Vinay Patil
> 
> On Tue, Oct 4, 2016 at 3:58 AM, Kostas Kloudas [via Apache Flink User Mailing List archive.]
<[hidden email] <x-msg://9/user/SendEmail.jtp?type=node&node=9323&i=0>>
wrote:
> Hello LF and Vinay,
> 
> With the introduction of “allowed lateness”  elements and windows are kept around
until the watermark
> passes the window.maxTimestamp + allowed_lateness and then they are cleaned up (garbage
collected)
> 
> Every element that comes in and belongs to a window that is garbage collected is dropped
as super-late.
> Elements that are late, but no more than the allowed lateness, they are kept the window
fires as before.
> 
> If you know what the maximum latency is, then the best way is to set the allowed lateness
to that value.
> 
> Currently Flink drops super-late elements and does not provide any mechanism to manually
handle these 
> elements, BUT there are discussions about adding such a functionality that will allow
you to (probably) have 
> a separate stream with only these elements.
> 
> Thanks,
> Kostas
> 
>> On Oct 3, 2016, at 5:28 PM, [hidden email] <http://user/SendEmail.jtp?type=node&node=9307&i=0>
wrote:
>> 
>> Not yet.
>> I'm hoping a Flink export on this mailing list will reply.
>> 
>> 
>> - LF
>> 
>> 
>> 
>> From: vinay patil <[hidden email] <http://user/SendEmail.jtp?type=node&node=9307&i=1>>
>> To: [hidden email] <http://user/SendEmail.jtp?type=node&node=9307&i=2>

>> Sent: Monday, October 3, 2016 8:09 AM
>> Subject: Re: Regarding Late Elements
>> 
>> Hi LF,
>> 
>> So did you manage to get the workaround for it ?
>> 
>> I am using a Custom Trigger which is similar to 1.0.3 with few changes
>> 
>> Regards,
>> Vinay Patil
>> 
>> On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List archive.]
<[hidden email] <>> wrote:
>> We have the same requirement - we cannot discard any data even if it arrives late.
>>  
>> 
>> - LF
>>  
>> 
>> 
>> 
>> From: Vinay Patil <[hidden email] <>>
>> To: [hidden email] <>
>> 
>> Sent: Sunday, October 2, 2016 8:21 PM
>> Subject: Regarding Late Elements
>> 
>> Hi Guys,
>> 
>> Just wanted to get an idea on Why Flink decided to completely discard late elements
in the latest version ?, this was not the case in 1.0.3
>> 
>> 
>> P.S In our case the data is critical so we cannot discard a single record even if
it is late, I have written a custom trigger (as suggested by Aljoscha) to even accept late
elements.
>> 
>> 
>> Regards,
>> Vinay Patil
>> 
>> 
>> 
>> 
>> If you reply to this email, your message will be added to the discussion below:
>> http://apache-flink-user- mailing-list-archive.2336050. n4.nabble.com/Regarding-Late-
Elements-tp9284p9292.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9292.html>
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
email] <> 
>> To unsubscribe from Apache Flink User Mailing List archive., click here <>.
>> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>> 
>> View this message in context: Re: Regarding Late Elements <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9294.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
at Nabble.com <http://nabble.com/>.
>> 
>> 
>> 
> 
> 
> 
> If you reply to this email, your message will be added to the discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9307.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9307.html>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
<x-msg://9/user/SendEmail.jtp?type=node&node=9323&i=1> 
> To unsubscribe from Apache Flink User Mailing List archive., click here <applewebdata://73527FE2-B120-400C-8866-E2985C929827>.
> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> 
> View this message in context: Re: Regarding Late Elements <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9323.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
at Nabble.com.


Mime
View raw message