flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
Subject Re: Streaming window : count with timeout ?
Date Fri, 17 Jul 2015 15:39:07 GMT
You can implement an custom window policy (I guess this should be
flexible enough for your case).

See documentation:
https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html#policy-based-windowing

If you have further question after reading, just go ahead :)

-Matthias


On 07/17/2015 05:30 PM, LINZ, Arnaud wrote:
> Hello,
> 
>  
> 
> The data in my stream have a timestamp that may be slightly out of
> order, but I need to process the data in the proper order. To do this, I
> use a windowing function and sort the items in a flatMap.
> 
>  
> 
> However, the source may sometimes send data in “bulk batches” and
> sometimes “on the fly”. If I choose a time window, it will suits well
> the “on the fly” behavior but when processing bulks I may have too many
> elements to sort in the time interval specified.
> 
>  
> 
> If I choose a “count.of” window, I will process batches efficiently but
> I may need to wait forever in the “on the fly” behavior until the count
> is reached.
> 
>  
> 
> What I need is then a “count window with time out” or a “time window
> with max element” : I would like to specify both a max count and a max
> time to fit the source behavior.
> 
>  
> 
> Do you have any idea how I can do that ?
> 
>  
> 
> Best regards,
> 
> Arnaud
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses
> pièces jointes. Toute utilisation ou diffusion non autorisée est
> interdite. Si vous n'êtes pas destinataire de ce message, merci de le
> détruire et d'avertir l'expéditeur.
> 
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.


Mime
View raw message