Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 12303200BD3 for ; Tue, 22 Nov 2016 06:03:27 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 10E8B160B19; Tue, 22 Nov 2016 05:03:27 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0D2DF160AF9 for ; Tue, 22 Nov 2016 06:03:25 +0100 (CET) Received: (qmail 11896 invoked by uid 500); 22 Nov 2016 05:03:25 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 11882 invoked by uid 99); 22 Nov 2016 05:03:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Nov 2016 05:03:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5CD3818002D for ; Tue, 22 Nov 2016 05:03:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.43 X-Spam-Level: * X-Spam-Status: No, score=1.43 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id FQ2OD3Cq_KKj for ; Tue, 22 Nov 2016 05:03:22 +0000 (UTC) Received: from mail-oi0-f41.google.com (mail-oi0-f41.google.com [209.85.218.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 343505FC9C for ; Tue, 22 Nov 2016 05:03:22 +0000 (UTC) Received: by mail-oi0-f41.google.com with SMTP id v84so7169699oie.3 for ; Mon, 21 Nov 2016 21:03:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=aT87Wlz3qR/hnQ4Gn2JGv/NTIXvHijcw9nEbqhYNvQY=; b=I1APx0dTa60x0crDLA17uvb3aS7tTu32yK2gGzeBlLEGgtFyumpX0a5OnrfQXgoCW0 WLAaf34Occx7E2z5MYnUJ1BjZ6dQoGgcTKdtEwJngAVxl5zQpfwyZztdsOAmHGDC78ks VDfjoPOMxmUCY2Mfq01gmr59E9qPiKXsT1+znAiNFg5gG5GomISpLSlYen2dMeE2L64X fPLQaR3eo3ib6NARuCPY1MQePcgEy9+cpThs5G2JKyXcKw9/NK9rNCHTZ5eNXBOYUt0z vkX0Y3Zliyq2bfyC+GM5TQdhmXUVWC82HgDHqH9AatxrOXYZzuwXIEHN2LW9zgCp7gtY dKKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=aT87Wlz3qR/hnQ4Gn2JGv/NTIXvHijcw9nEbqhYNvQY=; b=SzKFD9bESRmaBVMPUQjKLAVb6QA12Mu2R1UBcpDDmfPu47tu5ahKyUlca+psNyoWu3 ks/qTo2f1Po4IzEErfPjUYjR9mLUJ6tZ/Ww/s3YlKd+hxTFOpIQJvFQ3Y9OpsCT9NcMv iIFzl4qLFn+F3fHfMHS+/jWEZEJdLeKGKnqAsiwVphHj7hNSDoG6Wb5MuUMWHI9b8m8I 084PaVB7rKX7jrR4r21fk+OMXhfSrSzoo4ZjuOW2s2GHMYxTMJ+uxL1rlzf6LOK3NQwC HVy9xVdLJVtcRaDMd/dq4B6xHNoAXhQ4lP/BbkyR+51s8CBkEVAW17jS8WQILeVTBjal sxVA== X-Gm-Message-State: AKaTC02dtfglQ05hxcHjFTsSs4wh1WUgZyyVBdaltfxkGuSewHt/yeCOtg29M4283HtfDaRJfYXlXY0o9tGTaA== X-Received: by 10.202.244.214 with SMTP id s205mr10328014oih.70.1479790998039; Mon, 21 Nov 2016 21:03:18 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.81.102 with HTTP; Mon, 21 Nov 2016 21:03:17 -0800 (PST) In-Reply-To: References: From: =?UTF-8?B?SnVhbiBSb2Ryw61ndWV6IEhvcnRhbMOh?= Date: Mon, 21 Nov 2016 21:03:17 -0800 Message-ID: Subject: Re: Early events To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a1134fec2c7cac70541dcb0b1 archived-at: Tue, 22 Nov 2016 05:03:27 -0000 --001a1134fec2c7cac70541dcb0b1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable That makes sense, thanks for your answer. Greetings, Juan On Mon, Nov 21, 2016 at 9:11 AM, Aljoscha Krettek wrote: > Hi, > yes, Flink is expected to buffer those until the watermark catches up wit= h > their timestamp. > > Cheers, > Aljoscha > > On Sun, 20 Nov 2016 at 06:18 Juan Rodr=C3=ADguez Hortal=C3=A1 < > juan.rodriguez.hortala@gmail.com> wrote: > >> Hi, >> >> There was a bug in my code, I was assigning the timestamps wrong and tha= t >> is why it looked like early events where assigned processing time. >> Surprisingly enought my test works both ok with early events. In fact I >> have modified my test data generator to generate early events or late >> events, and both seem to work ok with my test (https://github.com/juanrh= / >> flink-state-eviction/blob/293fe1cf972b2e4bc6fb4e874eb8ba >> 70c78f7894/src/main/java/com/github/juanrh/streaming/source/ >> EventTimeDelayedElementsSource.java, https://github.com/juanrh/ >> flink-state-eviction/blob/293fe1cf972b2e4bc6fb4e874eb8ba >> 70c78f7894/src/test/java/com/github/juanrh/streaming/source/ >> EventTimeDelayedElementsSourceTest.java) >> >> Anyway, is this the expected behaviour for early events? Is Flink >> buffering early events until their future timestamp arrives? >> >> Thanks, >> >> Juan >> >> >> On Sat, Nov 19, 2016 at 8:31 PM, Juan Rodr=C3=ADguez Hortal=C3=A1 < >> juan.rodriguez.hortala@gmail.com> wrote: >> >> Hi, >> >> Maybe this is already in the documentation, sorry if I'm asking somethin= g >> obvious. I was thinking that if you have event time then you can also ha= ve >> early events, which would be events whose extracted timestampt is in the >> future. This might happen in practice for example in sensors with a skew= ed >> clock, that assign timestamps in the future to the events. I have made a >> simple test with a time window (https://github.com/juanrh/ >> flink-state-eviction/commit/09c2c1fe1e6068b0703c0833b8a574313cdca5a2), >> and it looks like Flink treats early events like events generated at the >> current processing time. What it's the expected behaviour of Flink for >> early events? >> >> Early events might be interesting for generating test data, if Flink was >> able to buffer those early events until its actual time arrives, althoug= h I >> guess implementing that would probably impact the performance in >> production. But as I say, early events might happen in production becaus= e >> you can have wrong clocks or wrong code in general in the devices that >> generate the events. Maybe a fallback to ingestion time would make sense= , >> and an approximation to that might be implemented with a timestamp >> extractor that overrides future timestamps with System.currenTimeMillis. >> >> Greetings, >> >> Juan >> >> >> --001a1134fec2c7cac70541dcb0b1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
That makes sense, thanks for your answer.
Greetings,

Juan
=
On Mon, Nov 21, 2016 at 9:11 AM, Aljoscha Kr= ettek <aljoscha@apache.org> wrote:
Hi,
yes, Flink is expected to buffer tho= se until the watermark catches up with their timestamp.

Cheers,
Aljoscha

On Sun, 20 Nov 201= 6 at 06:18 Juan Rodr=C3=ADguez Hortal=C3=A1 <juan.rodriguez.hortala@gmail.com> wrote:
Hi,

There was a bug in my code, I was assigning the= timestamps wrong and that is why it looked like early events where assigne= d processing time. Surprisingly enought my test works both ok with early ev= ents. In fact I have modified my test data generator to generate early even= ts or late events, and both seem to work ok with my test (https://github.com/juanrh/flink-state-eviction/blob/293fe1= cf972b2e4bc6fb4e874eb8ba70c78f7894/src/main/java/com/github/juanr= h/streaming/source/EventTimeDelayedElementsSource.java, = https://github.com/juanrh/flink-state-evic= tion/blob/293fe1cf972b2e4bc6fb4e874eb8ba70c78f7894/src/test/java/= com/github/juanrh/streaming/source/EventTimeDelayedElementsS= ourceTest.java)

Anyway, is this the expected b= ehaviour for early events? Is Flink buffering early events until their futu= re timestamp arrives?

Thanks,

Juan=


On Sat, Nov 19, 2016 at 8:31 PM, J= uan Rodr=C3=ADguez Hortal=C3=A1 <juan.rodriguez.hort= ala@gmail.com> wrote:
Hi,

Maybe t= his is already in the documentation, sorry if I'm asking something obvi= ous. I was thinking that if you have event time then you can also have earl= y events, which would be events whose extracted timestampt is in the future= . This might happen in practice for example in sensors with a skewed clock,= that assign timestamps in the future to the events. I have made a simple t= est with a time window (https://github.com/juanrh/flin= k-state-eviction/commit/09c2c1fe1e6068b0703c0833b8a574313cdca5a2<= /a>), and it looks like Flink treats early events like events generated at = the current processing time. What it's the expected behaviour of Flink = for early events?

Early events might be interesting for gene= rating test data, if Flink was able to buffer those early events until its = actual time arrives, although I guess implementing that would probably impa= ct the performance in production. But as I say, early events might happen i= n production because you can have wrong clocks or wrong code in general in = the devices that generate the events. Maybe a fallback to ingestion time wo= uld make sense, and an approximation to that might be implemented with a ti= mestamp extractor that overrides future timestamps with System.currenTimeMi= llis.

Greetings,

Juan


--001a1134fec2c7cac70541dcb0b1--