flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: How can i merge more than one flink stream
Date Tue, 25 Jul 2017 06:40:37 GMT
What do you mean by "consistent"? 
Of course you can do this only at the time the timpstamp is defined (e.g. Using NTP). However,
this is never perfect .
Then it is unrealistic that they always end up in the same window because of network delays
etc. you will need here a global state that is defined based on your use case (why do you
need this?)

> On 25. Jul 2017, at 08:35, Jone Zhang <joyoungzhang@gmail.com> wrote:
> 
> Thanks for your reply. I have another question:
> In my situation, each of the three streams contains a local timestamp segment. How can
I ensure that their timestamps are consistent in each time window before the merging operation?
And how to ensure the arrival of all the streams with consistent timestamps in each time window?
> 
> Thanks.
> 
> 2017-07-20 13:39 GMT+08:00 Kien Truong <duckientruong@gmail.com>:
>> Hi, 
>> 
>> To expand on Fabian's answer, there's a few API for join. 
>> 
>> * connect - you have to provide a CoprocessFunction. 
>> 
>> * window join/cogroup - you provide  key selector functions, a time window and a
join/cogroup function. 
>> 
>> With the first method, you have to write more code, in exchange for much more flexible
join condition. 
>> 
>> Regards, 
>> Kien 
>> 
>>> On Jul 20, 2017, at 01:55, Fabian Hueske <fhueske@gmail.com> wrote:
>>> Hi, 
>>> 
>>> there are basically two operations to merge streams. 
>>> 
>>> 1. Union simply merges the input streams such that the resulting stream has the
records of all input streams. Union is a built-in operator in the DataStream API. For that
all streams must have the same data type. 
>>> 2. Join connects records of streams according to a join condition. When joining
streams, this condition is often based on some time bounds. Join usually needs to be manually
implemented using a stateful CoProcessFunction. 
>>> 
>>> Once the streams are unioned or joined, you can apply a time-window on the result
stream. 
>>> 
>>> Best, Fabian 
>>> 
>>> 2017-07-19 9:05 GMT+02:00 Jone Zhang    <joyoungzhang@gmail.com>: 
>>>> I have three data streams
>>>> 1. app exposed and click
>>>> 2. app download
>>>> 3. app install
>>>> 
>>>> How can i merge the streams to create a unified stream,then compute it on
time-based windows 
>>>> 
>>>> Thanks
>>> 
> 

Mime
View raw message