flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominik Safaric <dominiksafa...@gmail.com>
Subject Re: Flink 1.2 time window operation
Date Thu, 30 Mar 2017 19:06:41 GMT
> First, some remarks here -  sources (in your case the Kafka consumer) will not stop fetching
/ producing data when the windows haven’t fired yet.
> 

This is for sure true. However, the plot shows the number of records produced per second,
where each record was assigned a created at timestamp while being created and before being
pushed back to Kafka. Sorry I did not clarify this before. Anyway, because of this I would
expect to have a certain lag. Of course, messages will not only be produced into Kafka exactly
at window expiry and then the produced shutdown - however, what concerns me is that messages
were produced to Kafka before the first window expired - hence the questions. 

> If you’re writing the outputs of the window operation to Kafka (by adding a Kafka sink
after the windowing), then yes it should only write to Kafka when the window has fired.


Hence, I this behaviour that you’ve described and we’ve expected did not occur. 

If it would help, I can share the source code and a detail Flink configuration. 

Cheers,
Dominik

> On 30 Mar 2017, at 13:09, Tzu-Li (Gordon) Tai <tzulitai@apache.org> wrote:
> 
> Hi,
> 
> Thanks for the clarification.
> 
>> What are the reasons behind consuming/producing messages from/to Kafka while the
window has not expired yet?
> 
> First, some remarks here -  sources (in your case the Kafka consumer) will not stop fetching
/ producing data when the windows haven’t fired yet. Does this explain what you have plotted
in the diagram you attached (sorry, I can’t really reason about the diagram because I’m
not so sure what the values of the x-y axes represent)?
> 
> If you’re writing the outputs of the window operation to Kafka (by adding a Kafka sink
after the windowing), then yes it should only write to Kafka when the window has fired. The
characteristics will also differ for different types of windows, so you should definitely
take a look at the Windowing docs [1] about them.
> 
> Cheers,
> Gordon
> 
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#window-assigners
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#window-assigners>
> On March 30, 2017 at 2:37:41 PM, Dominik Safaric (dominiksafaric@gmail.com <mailto:dominiksafaric@gmail.com>)
wrote:
> 
>> What are the reasons behind consuming/producing messages from/to Kafka while the
window has not expired yet?


Mime
View raw message