Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CANwfUVOAjcegTZYj-YJ0dUwMjb5XLyHH8yK1duhrbBiNo24tZQ@mail.gmail.com>
References: <CANwfUVOAjcegTZYj-YJ0dUwMjb5XLyHH8yK1duhrbBiNo24tZQ@mail.gmail.com>
From: Xingcan Cui <xingcanc@gmail.com>
Date: Thu, 9 Nov 2017 01:12:02 +0800
Message-ID: <CALN2whg=+4U7xN1F8QYJaL2Y5tnyLkqt4EY4gPN0ezPJmMs4kA@mail.gmail.com>
Subject: Re: Generate watermarks per key in a KeyedStream
To: Shailesh Jain <shailesh.jain@stellapps.com>
Cc: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="f403045c56daac0fed055d7bcbce"
archived-at: Wed, 08 Nov 2017 17:12:49 -0000

--f403045c56daac0fed055d7bcbce
Content-Type: text/plain; charset="UTF-8"

Hi Shailesh,

actually, the watermarks are generated per partition, but all of them will
be forcibly aligned to the minimum one during processing. That is decided
by the semantics of watermark and KeyedStream, i.e., the watermarks belong
to a whole stream and a stream is made up of different partitions (one per
key).

If the physical devices work in different time systems due to delay, the
event streams from them should be treated separately.

Hope that helps.

Best,
Xingcan

On Wed, Nov 8, 2017 at 11:48 PM, Shailesh Jain <shailesh.jain@stellapps.com>
wrote:

> Hi,
>
> I'm working on implementing a use case wherein different physical devices
> are sending events, and due to network/power issues, there can be a delay
> in receiving events at Flink source. One of the operators within the flink
> job is the Pattern operator, and there are certain patterns which are time
> sensitive, so I'm using Event time characteristic. But the problem comes
> when there are unpredictable delays in events from a particular device(s),
> which causes those events to be dropped (as I cannot really define a static
> bound to allow for lateness).
>
> Since I'm using a KeyedStream, keyed on the source device ID, is there a
> way to allow each CEP operator instance (one per key) to progress its time
> based on the event time in the corresponding stream partition. Or in other
> words, is there a way to generate watermarks per partition in a KeyedStream?
>
> Thanks,
> Shailesh
>

--f403045c56daac0fed055d7bcbce
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:arial,he=
lvetica,sans-serif">Hi Shailesh,</div><div class=3D"gmail_default" style=3D=
"font-family:arial,helvetica,sans-serif"><br></div><div class=3D"gmail_defa=
ult" style=3D"font-family:arial,helvetica,sans-serif">actually, the waterma=
rks are generated per partition, but all of them will be forcibly aligned t=
o the minimum one during processing. That is decided by the semantics of wa=
termark and KeyedStream, i.e., the watermarks belong to a whole stream and =
a stream is made up of different partitions (one per key).</div><div class=
=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif"><br></d=
iv><div class=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-s=
erif">If the physical devices work in different time systems due to delay, =
the event streams from them should be treated separately.</div><div class=
=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif"><br></d=
iv><div class=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-s=
erif">Hope that helps.</div><div class=3D"gmail_default" style=3D"font-fami=
ly:arial,helvetica,sans-serif"><br></div><div class=3D"gmail_default" style=
=3D"font-family:arial,helvetica,sans-serif">Best,</div><div class=3D"gmail_=
default" style=3D"font-family:arial,helvetica,sans-serif">Xingcan</div></di=
v><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Nov 8, =
2017 at 11:48 PM, Shailesh Jain <span dir=3D"ltr">&lt;<a href=3D"mailto:sha=
ilesh.jain@stellapps.com" target=3D"_blank">shailesh.jain@stellapps.com</a>=
&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0=
 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>=
<div><div><div>Hi,<br><br></div>I&#39;m working on implementing a use case =
wherein different physical devices are sending events, and due to network/p=
ower issues, there can be a delay in receiving events at Flink source. One =
of the operators within the flink job is the Pattern operator, and there ar=
e certain patterns which are time sensitive, so I&#39;m using Event time ch=
aracteristic. But the problem comes when there are unpredictable delays in =
events from a particular device(s), which causes those events to be dropped=
 (as I cannot really define a static bound to allow for lateness).<br><br><=
/div>Since I&#39;m using a KeyedStream, keyed on the source device ID, is t=
here a way to allow each CEP operator instance (one per key) to progress it=
s time based on the event time in the corresponding stream partition. Or in=
 other words, is there a way to generate watermarks per partition in a Keye=
dStream?<br><br></div>Thanks,<br></div>Shailesh<br></div>
</blockquote></div><br></div>

--f403045c56daac0fed055d7bcbce--