From user-return-27409-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Apr 29 09:46:07 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9F2CD18061A for ; Mon, 29 Apr 2019 11:46:06 +0200 (CEST) Received: (qmail 81929 invoked by uid 500); 29 Apr 2019 09:41:40 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 81839 invoked by uid 99); 29 Apr 2019 09:41:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Apr 2019 09:41:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id AD9C51805E5 for ; Mon, 29 Apr 2019 09:41:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.112 X-Spam-Level: *** X-Spam-Status: No, score=3.112 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 9ZRLXh6tZWcO for ; Mon, 29 Apr 2019 09:41:35 +0000 (UTC) Received: from mail-oi1-f194.google.com (mail-oi1-f194.google.com [209.85.167.194]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 976825FB67 for ; Mon, 29 Apr 2019 09:41:35 +0000 (UTC) Received: by mail-oi1-f194.google.com with SMTP id t70so2212781oif.5 for ; Mon, 29 Apr 2019 02:41:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=EwtShuDGvfsque4Ty6mBa0rQQTt0gOyqhMVgobWTRUY=; b=eDHbZKD6XU7dHizbIcVHsQCNITq8fN7ininh/xVmrPz7qkZbilHsiV6NyXPtpbTCqw RFvErz8GlDddi3ReGVfKt9SVTk8jwsAAS0KVq1UdwXgAwveX6nVW0huHQM/d8LJii/pD uETeQTV/i4dU1Pna6Y8sXTVN/5ho0f0w1uyaJ3mME1ZHSZ1wR6+CYbc7mh7/txTopO99 ibVcPxmbTpNHVR0NYR+lryIM2QdhLKXc9+BLqd5UAk0x00Cy9sAmamUsIz53zhIXzc0W UlvyidQK9ElRLIjTJ5F3tcOuEbHoIcQdPzH59zSYtj/p2hRzEudI4zgW3grFFesnHMaS d8ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=EwtShuDGvfsque4Ty6mBa0rQQTt0gOyqhMVgobWTRUY=; b=uIU5QkPS29q3uiye+FpVIKf5xxH/k/VpQDsQojDXMisBlnG0Pf8lEtOMLHnw1X5/vL wwkHtDfjbsZmesrA5qgFpV/VcqnmN11coA2zyaH9jvebl8gkxFYykNdbJG8YIre3vWFt uO1nmP6M1vlHiZ/D5FMEkNrgqFS7OMrwe4wQdAs0xNtqyJePAW6Po1vkEX8OQGQPBLB2 +FtBJ1K3/V1gW6QN/Z1MlU8DljPnXqD/GDmgSawCSKMwQ5Ezq9rummFHBzimANb+TvD3 AJXYJB50g2AQ9wY+/tMtdfJkkpTkHReAP0BmikWOCNGjAr1IdnJHYd3FK8HFEDb8Dd9S KAsA== X-Gm-Message-State: APjAAAUMSEPN3xUmPV8DfDB9L61C6y7t/bwp8a7+NT514eUrL1C/FHU0 iukeiHPeijqGmcBWNrzFFteykHVEo2uTgLIchnQ= X-Google-Smtp-Source: APXvYqzMlWP6z0RfLQ0ZqUEF80P34PdfFnpMRbFuYDrismzjW+sEQnYt7zzj7rCl7+DVL8RMgmG2nH9Ipg6DbVLSeL4= X-Received: by 2002:aca:df09:: with SMTP id w9mr14396672oig.94.1556530895001; Mon, 29 Apr 2019 02:41:35 -0700 (PDT) MIME-Version: 1.0 References: <773533722.4480718.1556032201464.ref@mail.yahoo.com> <773533722.4480718.1556032201464@mail.yahoo.com> <1026555245.4968257.1556106443722@mail.yahoo.com> <1594413804.1555986.1556461871643@mail.yahoo.com> In-Reply-To: <1594413804.1555986.1556461871643@mail.yahoo.com> From: Fabian Hueske Date: Mon, 29 Apr 2019 11:40:58 +0200 Message-ID: Subject: Re: Apache Flink - Question about dynamically changing window end time at run time To: M Singh Cc: Sameer W , Rong Rong , User Content-Type: multipart/alternative; boundary="000000000000141ebc0587a8170c" --000000000000141ebc0587a8170c Content-Type: text/plain; charset="UTF-8" Hi Mans, I don't know if that would work or not. Would need to dig into the source code for that. TBH, I would recommend to check if you can implement the logic using a (Keyed-)ProcessFunction. IMO, process functions are a lot easier to reason about than Flink's windowing framework. You can manage state and timer all by yourself and make sure everything is properly cleaned up. Best, Fabian Am So., 28. Apr. 2019 um 16:31 Uhr schrieb M Singh : > Thanks Sameer/Rong: > > As Fabian and you have mentioned, the window still sticks around forever > for global window, so I am trying avoid that scenario. > > Fabian & Flink team - do you have any insights into what would happen if I > create a window and the later change it's end time during the stream > processing ? Would it mess up any internal state/processing that uses the > end time when the window was first created ? If there is any other > consideration to keep in mind, please let me know. > > Thanks again. > > On Wednesday, April 24, 2019, 1:29:18 PM EDT, Rong Rong < > walterddr@gmail.com> wrote: > > > Hi Mans, > > Sameer is correct. if you would like to control window triggering based on > other elements that does not belong to this window (in a keyed stream > context) then this is probably the best way to approach. > > I think you've also posted in another thread that describes what will be > left after fire-and-purge [1]. As Fabian stated: the only thing that > might've left after is the window (which is the 2 long values indicate the > start/end) and the trigger object. But you are right it might eventually > filled up memory. > > Another approach is to implement your own operator that handles all these > internally by your user code. This would require you to replicate many of > the window operator logic though. > > Thanks, > Rong > > [1] > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-How-to-destroy-global-window-and-release-it-s-resources-td27191.html#a27212 > > On Wed, Apr 24, 2019 at 5:02 AM Sameer W wrote: > > Global Windows is fine for this use case. I have used the same strategy. > You just define custom evictors and triggers and you are all good. Windows > are managed by keys, so as such as long as events are evicted from the > window, that counts towards reclaiming memory for the key+window > combination. Plus there is just window per key with Global Windows. > > On Wed, Apr 24, 2019 at 7:47 AM M Singh wrote: > > Hi Rong: > > Thanks for your answer. > > From what I understand the dynamic gap session windows are also created > when the event is encountered. I need to be able to change the window end > time at a later time based on what other events are in that window. One > way to do this is to use GlobalWindows but then these are never deleted. > > Regarding CEP option - I believe that CEP patterns cannot be changed > dynamically once they've been complied which limits it usage. > > Please feel free to correct me. > > Thanks for your help and pointers. > > On Tuesday, April 23, 2019, 8:12:56 PM EDT, Rong Rong > wrote: > > > Hi Mans, > > I am not sure what you meant by "dynamically change the end-time of a > window. If you are referring to dynamically determines the firing time of > the window, then it fits into the description of session window [1]: > If you want to handle window end time dynamically, one way of which I can > think of is the dynamic gap, session window [1] approach. with which you > can specify the end-time of a window based on input elements. Provided that > you are maintaining a session window. > Another way to look at it is through the Flink-CEP library [2]. > > Thanks, > Rong > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/windows.html#session-windows > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/libs/cep.html#groups-of-patterns > > On Tue, Apr 23, 2019 at 8:19 AM M Singh wrote: > > Hi: > > I am working on a project and need to change the end time of the window > dynamically. I want to find out if the end time of the window is used > internally (for sorting windows/etc) except for handling watermarks that > would cause problems if the end time was changed during run time after the > window has been created even if no new event has arrived for that window. > > I don't want to use GlobalWindow since from my understanding it never gets > destroyed. > > If there is any alternate way of dealing with this, please let me know. > > Thanks > > Mans > > --000000000000141ebc0587a8170c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Mans,

I don't know if= that would work or not. Would need to dig into the source code for that. <= br>

TBH, I would recommend to check if you can imp= lement the logic using a (Keyed-)ProcessFunction.
IMO, process fu= nctions are a lot easier to reason about than Flink's windowing framewo= rk.
You can manage state and timer all by yourself and make = sure everything is properly cleaned up.

Best,<= /div>
Fabian


Am So., 28. Apr. 2019 um 16:31=C2= =A0Uhr schrieb M Singh <mans2singh@yahoo.com>:
Thanks Sameer/Rong:

As Fabian and you= have mentioned, the window still sticks around forever for global window, = so I am trying avoid that scenario.

Fabian & F= link team - do you have any insights into what would happen if I create a w= indow and the later change it's end time during the stream processing ?= =C2=A0 Would it mess up any internal state/processing that uses the end tim= e when the window was first created ?=C2=A0 If there is any other considera= tion to keep in mind, please let me know.

Thanks a= gain.

=20
=20
On Wednesday, April 24, 2019, 1:29:18 PM EDT, Rong Rong= <walterddr@gma= il.com> wrote:


Hi Mans,

Sameer is correct. if you would like to control window tri= ggering based on other elements that does not belong to this window (in a k= eyed stream context) then this is probably the best way to approach.=C2=A0<= /div>

I think you've also posted in a= nother thread that describes what will be left after fire-and-purge [1]. As= Fabian stated: the only thing that might've left after is the window (= which is the 2 long values indicate the start/end) and the trigger object. = But you are right it might eventually filled up memory.

Another approach is to implement your own operator tha= t handles all these internally by your user code. This would require you to= replicate many of the window operator logic though.

Thanks,
Rong

On Wed, Apr 24, 2= 019 at 5:02 AM Sameer W <sameer@axiomine.com> wrote= :
Global Windows is fine for this use case. I have used the same s= trategy. You just define custom evictors and triggers and you are all good.= Windows are managed by keys, so as such as long as events are evicted from= the window, that counts towards reclaiming memory for the key+window combi= nation. Plus there is just window per key with Global Windows.=C2=A0
<= br clear=3D"none">
On Wed= , Apr 24, 2019 at 7:47 AM M Singh <mans2singh@yahoo.com> wrote:
Hi Rong:

Thanks for your ans= wer.

From what I understand the dyn= amic gap session windows are also created when the event is encountered.=C2= =A0 I need to be able to change the window end time at a later time based o= n what other events are in that window.=C2=A0 One way to do this is to use = GlobalWindows but then these are never deleted.

Regarding CEP option - I believe that CEP p= atterns cannot be changed dynamically once they've been complied which = limits it usage.

Please feel= free to correct me.

Thanks for your help and pointers.

=20
=20


Hi Mans,

=
I am not sure what you meant by "dynamically change the end= -time of a window. If you are referring to dynamically determines the firin= g time of the window, then it fits into the description of session window [= 1]:=C2=A0
If you want to handle window end time dynamically, one = way of which I can think of is the dynamic gap, session window [1] approach= . with which you can specify the end-time of a window based on input elemen= ts. Provided that you are maintaining a session window.=C2=A0
Ano= ther way to look at it is through the Flink-CEP library [2].=C2=A0

Thanks,
Rong



On Tue, Apr 23, 2019 at 8:19 AM M S= ingh <mans2singh@yahoo.com> wrote:
Hi:

I am= working on a project and need to change the end time of the window dynamic= ally.=C2=A0 I want to find out if the end time of the window is used intern= ally (for sorting windows/etc) except for handling watermarks that would ca= use problems if the end time was changed during run time after the window h= as been created even if no new event has arrived for that window.

I don't want to use = GlobalWindow since from my understanding it never gets destroyed.

If there is any alternat= e way of dealing with this, please let me know.

Thanks

<= div>Mans
--000000000000141ebc0587a8170c--