flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Combine two independant streams
Date Tue, 28 Feb 2017 16:19:30 GMT
Hi Markus,

I'm not sure I understood the issue with the second approach.
Is it that the stream of application events might be empty for some time
such that its event time is not increasing?

Best, Fabian

2017-02-28 17:02 GMT+01:00 Markus Klein <klein.markus90@gmx.de>:

> Hello Flink Community,
>
> I have a question regarding combining two independant streams.
> The first stream is a stream of events with metrics information. It occurs
> every 10 seconds. What I want is to join a second stream with events from
> an application. The result should be an event with the metrics and the
> events that happened the last 10 seconds.
>
> So my first approacch was to generate an ID which will be increased after
> every metric event. This ID will be added to the application events and of
> cours for the current metric event. This works somehow good for live events
> but for recalculating past events the two streams have to start at the same
> point in event time.
>
> The second approach was to generate a time window of 10 seconds for the
> application events and for the metrics and set the window end time as a key
> because flink windows end at e.g. 11:01:10, then 11:01:20 and so on. But
> this approach works only for past events because flink needs another
> application event to know that 10 seconds have passed for the application
> events window.
>
> I hope you guys understand the problem. Is there a way two combine them in
> a nice way? I don't want to generate empty "heartbeats" for the application
> event stream.
>
> Thanks for your help.
>
> Greetings
> Markus
>

Mime
View raw message