Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0AD5D160C15 for ; Wed, 3 Jan 2018 15:36:22 +0100 (CET) Received: (qmail 5196 invoked by uid 500); 3 Jan 2018 14:36:17 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 5186 invoked by uid 99); 3 Jan 2018 14:36:16 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Jan 2018 14:36:16 +0000 Received: from aljoschas-mbp.fritz.box (ip-2-205-86-231.web.vodafone.de [2.205.86.231]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 1BE0E1A00AE; Wed, 3 Jan 2018 14:36:15 +0000 (UTC) From: Aljoscha Krettek Message-Id: <92A59A4A-6509-4B1A-AC97-350F43BEC2D7@apache.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_1F6C4DBC-9DD1-4C8C-BD51-8B2E4605FB91" Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: Flink CEP with event time Date: Wed, 3 Jan 2018 15:36:12 +0100 In-Reply-To: Cc: user To: shashank agarwal References: X-Mailer: Apple Mail (2.3445.5.20) archived-at: Wed, 03 Jan 2018 14:36:23 -0000 --Apple-Mail=_1F6C4DBC-9DD1-4C8C-BD51-8B2E4605FB91 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Ok, but will there be events in all Kafka partitions/topics? > On 3. Jan 2018, at 15:33, shashank agarwal = wrote: >=20 > Hi, >=20 > Yes, Events will always carry a variable OriginTimestamp which I am = using in the extractor. I have used fallback also in case of data = missing will put System current millis.=20 >=20 > Still, it's not printing results. >=20 > Best, > Shashank >=20 >=20 >=20 >=20 >=20 > =E2=80=8C >=20 > On Wed, Jan 3, 2018 at 7:40 PM, Aljoscha Krettek > wrote: > Hi, >=20 > Are all the partitions always carrying data that has advancing = timestamps? When using Event-time the Kafka source (and Flink in = general) needs to have steady progress in all partitions, otherwise the = watermark does not advance, which in turn means that processing will be = stalled downstream. >=20 > Best, > Aljoscha >=20 >=20 >> On 3. Jan 2018, at 14:29, shashank agarwal > wrote: >>=20 >> Hello, >>=20 >> I have some patterns in my program. For an example, >>=20 >> A followedBy B. >>=20 >> As I am using kafka source and my event API's using load balancers so = sometimes B comes before A. So my CEP doesn't generate any result for = those events.=20 >>=20 >> I have then tried event time and applied = "BoundedOutOfOrdernessTimestampExtractor" on kafkasource with extract = time from an origin time variable which I have in the event. I am using = watermark lateness of 10 seconds in that. >>=20 >> Now CEP stopped generating results. It's not even generating results = where Event B comes after A. I have tried within (10 seconds) in CEP = also still not generating results.=20 >>=20 >> Am I doing anything wrong? >>=20 >> I have to cover the case where B can come after A from Kafka. >>=20 >> --=20 >> Thanks Regards >>=20 >> SHASHANK AGARWAL >> --- Trying to mobilize the things.... >>=20 >>=20 >>=20 >>=20 >> =E2=80=8C >=20 >=20 >=20 >=20 > --=20 > Thanks Regards >=20 > SHASHANK AGARWAL > --- Trying to mobilize the things.... --Apple-Mail=_1F6C4DBC-9DD1-4C8C-BD51-8B2E4605FB91 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Ok, = but will there be events in all Kafka partitions/topics?

On 3. Jan 2018, at 15:33, shashank agarwal <shashank734@gmail.com> wrote:

3D""Hi,

Yes, Events will always carry a = variable OriginTimestamp which I am using in the extractor. I have used = fallback also in case of data missing will put System current = millis. 

Still, it's not printing results.

Best,
Shashank





=E2=80=8C

On Wed, Jan 3, 2018 at 7:40 PM, = Aljoscha Krettek <aljoscha@apache.org> wrote:
Hi,

Are = all the partitions always carrying data that has advancing timestamps? = When using Event-time the Kafka source (and Flink in general) needs to = have steady progress in all partitions, otherwise the watermark does not = advance, which in turn means that processing will be stalled = downstream.

Best,
Aljoscha


On 3. Jan 2018, at 14:29, = shashank agarwal <shashank734@gmail.com> = wrote:

3D""Hello,

I have some patterns in my program. For = an example,

 A followedBy B.

As I am using kafka source and my event = API's using load balancers so sometimes B comes before A. So my CEP = doesn't generate any result for those events. 

I have then tried event = time and applied "BoundedOutOfOrdernessTimestampExtractor" on kafkasource with extract time from an origin = time variable which I have in the event. I am using watermark lateness = of 10 seconds in that.

Now CEP stopped generating results. It's not even generating = results where Event B comes after A. I have tried within (10 seconds) in = CEP also still not generating results. 

Am I doing anything wrong?

I have to cover = the case where B can come after A from Kafka.

-- 
Thanks = Regards

SHASHANK AGARWAL
 ---  Trying to mobilize the = things....




=E2=80=8C


-- 
Thanks = Regards

SHASHANK AGARWAL
 ---  Trying to mobilize the = things....

= --Apple-Mail=_1F6C4DBC-9DD1-4C8C-BD51-8B2E4605FB91--