From user-return-30767-archive-asf-public=cust-asf.ponee.io@flink.apache.org Thu Nov 7 13:11:54 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id BD19C180630 for ; Thu, 7 Nov 2019 14:11:53 +0100 (CET) Received: (qmail 84180 invoked by uid 500); 7 Nov 2019 13:11:51 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 84170 invoked by uid 99); 7 Nov 2019 13:11:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Nov 2019 13:11:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id AE7B8180A6D for ; Thu, 7 Nov 2019 13:11:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id m1ooSlVvzOzV for ; Thu, 7 Nov 2019 13:11:49 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::335; helo=mail-ot1-x335.google.com; envelope-from=victtim@gmail.com; receiver= Received: from mail-ot1-x335.google.com (mail-ot1-x335.google.com [IPv6:2607:f8b0:4864:20::335]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id C26547DDF1 for ; Thu, 7 Nov 2019 13:11:48 +0000 (UTC) Received: by mail-ot1-x335.google.com with SMTP id r24so1908633otk.12 for ; Thu, 07 Nov 2019 05:11:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=bBPfZF4M4/Mzo7mqFrmDlrUJyJvIXW80YpxlTdMPr+8=; b=P4jHdMVvnbBDLiPyw/O/01G/X1/dR6UQ2t81wj++uFmvYPr4hj534ABOlIRQsD423Y nFXdSQJHsA6hOEt8CT6R+KWoRrOIsp6Pp8HQ3ViBSvfDQTT/EVvnCdspZ//t/C8/leBA GFwDz+nO5M//NMokVCJIVYIZU9/PukSupiMThC3GIzWipp7zAfh2yDJWfbePHeprXWFU NYotFjaC1XcVKfPxSaEn4UKtJTkEql+iMXchOXhIi/BOUgYXN9tGhNpTK25hnJlAney0 pfsCq5H4zmo6rmSI3sWZxREnp2cj2l0pVaSYfR5xBN7xdxRG8f+W/bqv7DYowP5VqEGW 0bMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=bBPfZF4M4/Mzo7mqFrmDlrUJyJvIXW80YpxlTdMPr+8=; b=fV5FLXPuA2vCKIFmfUgLis4i3FxFYUJwMbrVcgyA/QJJD1IvSLygA8Nf9wSRgI5hjS m4/ezwwrHuxJVA7lhH8i6smOjTqvnBc+2wAKkaqCPqrYzeHf0lOq2hk2Q+sf9kCKgFGs R8NyffyIJqZGjc0YIzwvAD2Nompm4ZHmuuyGOv+0+2unkL9r7pFQDt4m23NiGi+xo8D7 4CRkxAWVkD7BjhqOb0ahvMm6u25ZzCLVhRoJYInHvredn1sUoqD63CG/PQdLGKq78Bil flOzz/eDrSYHSECtPLKzqClXMbEblxKGTPqtIOyB3vT42UUobSbkkXYDssZOcKBlBrqv YVoA== X-Gm-Message-State: APjAAAWtmL5DHZQw2GJwx0URdxfbcizd7wsftlkZXdyhn2uVfX5DImpN 6p5yCU5hJO4KxC+oS+z4/CAxFI07rqHDnki9HZ7XXQ== X-Google-Smtp-Source: APXvYqzwNNTZHz9OqMY91cWPJDzdSZC6owMMEeVNKEo1m6hZOH4na5OGhZws51GE24C/utOELu6vcolBXP05I0LNVHQ= X-Received: by 2002:a05:6830:4cf:: with SMTP id s15mr3036112otd.261.1573132306745; Thu, 07 Nov 2019 05:11:46 -0800 (PST) MIME-Version: 1.0 From: Timothy Victor Date: Thu, 7 Nov 2019 07:11:34 -0600 Message-ID: Subject: Flink Filters have state? To: user Content-Type: multipart/alternative; boundary="000000000000540c990596c168d5" --000000000000540c990596c168d5 Content-Type: text/plain; charset="UTF-8" I have a FilterFunction implementation which accepts an argument in its constructor which it stores as an instance member. For example: class ThresholdFilter implements FilterFunction { private final MyThreshold threshold; private int numElementsSeen; public ThresholdFilter(MyThreshold threshold) { this.threshold = threshold; } } The filter uses the threshold in deciding whether or not to filter the incoming element. All this works but I have some gaps in my understanding. 1. How is this filter stored and recovered in the case of a failure. Is it just serialized to a POJO and stored in the configured state backend? 2. When recovered will it maintain the state of all members (e.g. note that I have a numElementsSeen member in the filter which will keep incrementi for each element recevied). 3. Is this sort of thing even advisable for a filter? I'm guessing Filters are meant to be reusable across operator instances. In which case the state could be wrong after recovery? Thanks in advance Tim --000000000000540c990596c168d5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I have a FilterFunction implementation which accepts an a= rgument in its constructor which it stores as an instance member.=C2=A0 =C2= =A0 For example:

class Thresho= ldFilter implements FilterFunction=C2=A0 {

=C2=A0 private final MyThreshold threshold;

=C2=A0 private int numElementsSeen;<= /div>

=C2=A0 public ThresholdF= ilter(MyThreshold threshold) {
=C2=A0 =C2=A0 this.th= reshold =3D threshold;
=C2=A0 }

=C2=A0 <more code>

}

The filter uses the threshold in deciding whether or not to filte= r the incoming element.

= All this works but I have some gaps in my understanding.

1.=C2=A0 =C2=A0How is this filter stored a= nd recovered in the case of a failure.=C2=A0 =C2=A0Is it just serialized to= a POJO and stored in the configured state backend?
=
2.=C2=A0 When recovered will it maintain the st= ate of all members (e.g. note that I have a numElementsSeen member in the f= ilter which will keep incrementi for each element recevied).

3.=C2=A0 Is this sort of thing even a= dvisable for a filter?=C2=A0 I'm guessing Filters are meant to be reusa= ble across operator instances.=C2=A0 In which case the state could be wrong= after recovery?

Thanks = in advance

Tim
--000000000000540c990596c168d5--