flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: AW: Is watermark used by joining two streams
Date Mon, 31 Jul 2017 07:46:29 GMT
Hi,

@Wei: You can implement very different behavior using a CoProcessFunction.
However, if your operator is time-based, the logical time of the operator
will be the minimum time of both streams (time of the "slower" watermark).

@Vijay: I did not understand what your requirements are. Do you want to
join or merge streams? Those are two different things. This thread
discusses joins not merging.

Best,
Fabian

2017-07-31 4:24 GMT+02:00 G.S.Vijay Raajaa <gsvijayraajaa@gmail.com>:

> Hi Fabian,
>
> How do I order by the merge time. Let's say I merge the stream at T1. I
> wanted to drop T2 merge if T2 < T1. Now depending on the arrival of data
> from individual stream and the time at which the merge happens, they become
> out of order. Any thoughts will be really appreciated.
>
> Regards,
> Vijay Raajaa GS
>
> On Jul 31, 2017 1:14 AM, "wei" <jixian01@googlemail.com> wrote:
>
> Hello Fabian,
>
>
>
> thank you for your answer!
>
>
>
> Does it mean that the operator will wait until get two watermarks from the
> input streams and emits then the “slower” watermark?
>
>
>
> Best regards
>
> Wei
>
>
>
> *Von:* Fabian Hueske [mailto:fhueske@gmail.com]
> *Gesendet:* Sunday, July 30, 2017 11:17 AM
> *An:* xie wei
> *Cc:* user
> *Betreff:* Re: Is watermark used by joining two streams
>
>
>
> Periodic and punctuated watermarks only differ in the way that they are
> generated. Afterwards they are treated the same.
>
> An operator with two input streams will always sync its own watermarks to
> the watermarks of both input streams, i.e., to the "slower" watermark of
> both inputs.
>
> So if the left input says it is 12:14 and the right says it is 11:53, the
> operator will have a internal time of 11:53 and emit watermarks according
> to that time.
>
> Hope that helps,
>
> Fabian
>
>
>
>
>
> 2017-07-28 15:00 GMT+02:00 xie wei <jixian01@googlemail.com>:
>
> Hello,
>
> i want to join two streams based on event time window, every stream has
> its own watermark, one has priodic watermark and the other has punctuated
> watermark.
>
> are the watermarks used to trigger the join? if yes, which one and how is
> it used?
>
> Thank you and best regards
>
> Wei
>
>
>
>
>
>

Mime
View raw message