Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 454DD200C32 for ; Thu, 9 Mar 2017 21:10:01 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 43DA2160B75; Thu, 9 Mar 2017 20:10:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 648B2160B5F for ; Thu, 9 Mar 2017 21:10:00 +0100 (CET) Received: (qmail 54692 invoked by uid 500); 9 Mar 2017 20:09:59 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 54682 invoked by uid 99); 9 Mar 2017 20:09:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Mar 2017 20:09:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id E14D0C14BD for ; Thu, 9 Mar 2017 20:09:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Qm5YfjH1Xycf for ; Thu, 9 Mar 2017 20:09:57 +0000 (UTC) Received: from mail-ua0-f169.google.com (mail-ua0-f169.google.com [209.85.217.169]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5DC795F1EE for ; Thu, 9 Mar 2017 20:09:57 +0000 (UTC) Received: by mail-ua0-f169.google.com with SMTP id u30so94019675uau.0 for ; Thu, 09 Mar 2017 12:09:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=qd6TiSsZIOCY7pmNM39aV9srNYZEJuh8QbO3K/OyADg=; b=PfG+UfxzO1fpyXIkq7xUAUJ/MjKVqU642q8UTM4SFKaS18C2Wf12vEalUJaeo7AwVh hKI1gBkwFPu/PtoxZ+3VRhXgpBuQFlubSEH9xmNTBWR7FWK43ZkaYoiwmArfSEGkbCDL jgkaPKnYUhr7Hcmah18yEIEjaLGIF9w8spBAeI8TKKajt844vmIrfug0NYDSgbhtAK4z M2B6x8bZ4YSW90Tr1ii+Y9a0pnMEhaBA/GfXbIPYGpIYZOOwL6O/xtTeVGqjFv5QWhQC w77TrkZHcpVNgIN7UN6KzLKMd3ap5hUuKuk1bWoRz/MY12i4YPomFON7EJHjEQMx0MHG v9Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=qd6TiSsZIOCY7pmNM39aV9srNYZEJuh8QbO3K/OyADg=; b=NvgJIMRBn+D+WR+iqRn/VfwsWBQhMQo29JpgmUYOUwvBr88nnp1DA0avkunJbKs8aj jeDZqEHZfeN1Y8am2FCC2coSyYAynLlDiVHH/6/QlPmnS66xL/pngOHAcZQeerg+3nLe CVErSOLZLJUl5Qk1/Of0Sq7Yjyu0rQ5ImO+Ebe2PBaU+eTmbn4UZOOS+TRlKy/b5YaPm G1/uvnqO13EdvCPlhKTV9lJuMOnk1y7bn4FarVAxVr/zSU+KNPL02rolNhuup1ARt5iO kYpk8nrMwiSonkkSqqcFHGheTEmNcXBbWbNKzymnpKJoS/EM5ex5q4sKhAhWDIm1+CmL MKDw== X-Gm-Message-State: AMke39m0xO2J3iz8cgOL2SUALh4KxW/bND6CnjnEq9OciItqCUXj0eUv4TA3MUyU7bqYzSXuWAoed4wrA3TvKw== X-Received: by 10.176.25.29 with SMTP id v29mr7672269uag.157.1489090196766; Thu, 09 Mar 2017 12:09:56 -0800 (PST) MIME-Version: 1.0 Received: by 10.103.105.6 with HTTP; Thu, 9 Mar 2017 12:09:55 -0800 (PST) In-Reply-To: <1489071303.4178207.905863176.6D5216F0@webmail.messagingengine.com> References: <1489071303.4178207.905863176.6D5216F0@webmail.messagingengine.com> From: Johannes Schulte Date: Thu, 9 Mar 2017 21:09:55 +0100 Message-ID: Subject: Re: TTL for State Entries / FLINK-3089 To: user@flink.apache.org Content-Type: multipart/alternative; boundary=f403043792cc37a2e0054a51d4a7 archived-at: Thu, 09 Mar 2017 20:10:01 -0000 --f403043792cc37a2e0054a51d4a7 Content-Type: text/plain; charset=UTF-8 Hey Aljoscha, thank you for your reply. The amount and quality of response on this list are really great to see and a good way to learn. I will try this and see how this works out. Cheers, Johannes On Thu, Mar 9, 2017 at 3:55 PM, Aljoscha Krettek wrote: > Hi Johannes, > I think what you can do is not register a timer for every event but for > every key, with a certain granularity. When that timer fires you check > what you want to clean up for that key and maybe register another timer > for the future. This way, the size of your timer state is bounded by > your key cardinality and I think people have used Flink with > timers/windows with key cardinalities of several 100 millions. > > Best, > Aljoscha > > On Wed, Mar 8, 2017, at 14:37, Ufuk Celebi wrote: > > Looping in Aljoscha and Kostas who are the expert on this. :-) > > > > On Mon, Mar 6, 2017 at 6:06 PM, Johannes Schulte > > wrote: > > > Hi, > > > > > > I am trying to achieve a stream-to-stream join with big windows and are > > > searching for a way to clean up state of old keys. I am already using a > > > RichCoProcessFunction > > > > > > I found there is already an existing ticket > > > > > > https://issues.apache.org/jira/browse/FLINK-3089 > > > > > > but I have doubts that a registration of a timer for every incoming > event is > > > feasible as the timers seem to reside in an in-memory queue. > > > > > > The task is somewhat similar to the following blog post: > > > http://devblog.mediamath.com/real-time-streaming- > attribution-using-apache-flink > > > > > > Is the implementation of a custom window operator a necessity for > achieving > > > such functionality > > > > > > Thanks a lot, > > > > > > Johannes > > > > > > > --f403043792cc37a2e0054a51d4a7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hey Aljoscha,

thank you for your reply.= The amount and quality of response on this list are really great to see an= d a good way to learn.

I will try this and see how= this works out.

Cheers,

= Johannes

On = Thu, Mar 9, 2017 at 3:55 PM, Aljoscha Krettek <aljoscha@apache.org&g= t; wrote:
Hi Johannes,
I think what you can do is not register a timer for every event but for
every key, with a certain granularity. When that timer fires you check
what you want to clean up for that key and maybe register another timer
for the future. This way, the size of your timer state is bounded by
your key cardinality and I think people have used Flink with
timers/windows with key cardinalities of several 100 millions.

Best,
Aljoscha

On Wed, Mar 8, 2017, at 14:37, Ufuk Celebi wrote:
> Looping in Aljoscha and Kostas who are the expert on this. :-)
>
> On Mon, Mar 6, 2017 at 6:06 PM, Johannes Schulte
> <johannes.schulte@gma= il.com> wrote:
> > Hi,
> >
> > I am trying to achieve a stream-to-stream join with big windows a= nd are
> > searching for a way to clean up state of old keys. I am already u= sing a
> > RichCoProcessFunction
> >
> > I found there is already an existing ticket
> >
> > https://issues.apache.org/jira/brows= e/FLINK-3089
> >
> > but I have doubts that a registration of a timer for every incomi= ng event is
> > feasible as the timers seem to reside in an in-memory queue.
> >
> > The task is somewhat similar to the following blog post:
> > http://devb= log.mediamath.com/real-time-streaming-attribution-using-apache-fl= ink
> >
> > Is the implementation of a custom window operator a necessity for= achieving
> > such functionality
> >
> > Thanks a lot,
> >
> > Johannes
> >
> >

--f403043792cc37a2e0054a51d4a7--