Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E636C200D11 for ; Mon, 2 Oct 2017 12:46:37 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E4B2E1609EF; Mon, 2 Oct 2017 10:46:37 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 364011609DE for ; Mon, 2 Oct 2017 12:46:37 +0200 (CEST) Received: (qmail 31809 invoked by uid 500); 2 Oct 2017 10:46:36 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 31797 invoked by uid 99); 2 Oct 2017 10:46:36 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Oct 2017 10:46:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 89877194B1D for ; Mon, 2 Oct 2017 10:46:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.822 X-Spam-Level: X-Spam-Status: No, score=-0.822 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=citi.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id xHY5jM2GWjB6 for ; Mon, 2 Oct 2017 10:46:34 +0000 (UTC) Received: from mx0a-00123c01.pphosted.com (mx-a.mail.citi.com [67.231.145.106]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id C7D185FB57 for ; Mon, 2 Oct 2017 10:46:33 +0000 (UTC) Received: from pps.filterd (m0083710.ppops.net [127.0.0.1]) by mx0a-00123c02.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id v92Aep1Y046255 for ; Mon, 2 Oct 2017 10:46:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citi.com; h=from : to : subject : date : message-id : content-type : content-transfer-encoding : mime-version; s=maila; bh=F90sYtyHVtfiovuiyYWbkEQwj0mXWiGMsnzuXURpeI0=; b=GvIOs1oC7bXFBT66C70znDTq/HXl+Y30P0qxCy2p9Cq6KRFIQGJN2uNj0sHFqSBOi9fk HzBXKgjRz5+1R6M41F7fTpXCQpVVIyO7NgOcIPnlGJ7Ct04rMujnA8S5kneBJ1WTDRKM 6aKdQUBn4eLixgRiMDZTXEH1rY3xg1ZOxC7mapEMo9PV1m11Wb+bb2XVH6pHmvuhiKSO l0AWhQpprUG+V77hJRU513ZsFapqheRUQc7yYFqFLoicD51tBhdOZPGh3ERH+itGgboh A2crT/klGzjZpoWzXg57+CoVISLwUi8sQCcOaA7qLtDc/FKD1ca4Rbu4ll+6vOXoISfg vA== Received: from mail.citigroup.com (smtpoutbound.citigroup.com [192.193.220.23]) by mx0a-00123c02.pphosted.com with ESMTP id 2da46n3fwg-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Mon, 02 Oct 2017 10:46:27 +0000 Received: from imbhub-mw35.nam.nsroot.net (imbhub-mw35.nam.nsroot.net [144.215.196.153]) by smtpinbound.citigroup.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.2.2) with ESMTP id v92AkP0r020686 for ; Mon, 2 Oct 2017 10:46:26 GMT Received: from imbdlprt-sw02.nam.nsroot.net (imbdlprt-sw02.nam.nsroot.net [169.177.231.238]) by imbhub-mw35.nam.nsroot.net (Sentrion-MTA-4.3.1/Sentrion-MTA-4.2.2) with ESMTP id v92AkP4Y005298 for ; Mon, 2 Oct 2017 10:46:25 GMT Received: from imbdlpbuf-ru02.nam.nsroot.net (namdlpdimpgt04.nam.nsroot.net [169.193.226.13]) by imbdlprt-sw02.nam.nsroot.net (Sentrion-MTA-4.3.1/Sentrion-MTA-4.2.2) with ESMTP id v92AkOTC015089 for ; Mon, 2 Oct 2017 10:46:24 GMT Received: from EXLNIHT06.eur.nsroot.net (EXLNIHT06.eur.nsroot.net [169.182.86.191]) by imbdlpbuf-ru02.nam.nsroot.net (Sentrion-MTA-4.3.1/Sentrion-MTA-4.2.2) with ESMTP id v92AkH8v002576 for ; Mon, 2 Oct 2017 10:46:24 GMT Received: from EXLNHT18.eur.nsroot.net (169.182.85.47) by EXLNIHT06.eur.nsroot.net (169.182.86.191) with Microsoft SMTP Server (TLS) id 14.3.352.0; Mon, 2 Oct 2017 11:46:09 +0100 Received: from EXLNMB58.eur.nsroot.net ([169.254.2.3]) by EXLNHT18.eur.nsroot.net ([169.182.85.47]) with mapi id 14.03.0352.000; Mon, 2 Oct 2017 11:46:08 +0100 From: "Marchant, Hayden " To: "user@flink.apache.org" Subject: In-memory cache Thread-Topic: In-memory cache Thread-Index: AdM7a6EyK0/c0BncQPeSWy2qbgvApQ== Date: Mon, 2 Oct 2017 10:46:08 +0000 Message-ID: <99B8991EBC5DF841B18C71E56C02D4A010CD1897@EXLNMB58.eur.nsroot.net> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [169.182.87.208] x-wiganss: 01000000010017EXLNHT18.eur.nsroot.net ID0042<99B8991EBC5DF841B18C71E56C02D4A010CD1897@EXLNMB58.eur.nsroot.net> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-10-02_03:,, signatures=0 archived-at: Mon, 02 Oct 2017 10:46:38 -0000 We have an operator in our streaming application that needs to access 'refe= rence data' that is updated by another Flink streaming application. This re= ference data has about ~10,000 entries and has a small footprint. This refe= rence data needs to be updated ~ every 100 ms. The required latency for th= is application is extremely low ( a couple of milliseconds), and we are the= refore cautious of paying cost of I/O to access the reference data remotely= . We are currently examining 3 different options for accessing this referen= ce data: 1. Expose the reference data as QueryableState and access it directly from = the 'client' streaming operator using the QueryableState API 2. same as #1, but create an In-memory Java cache of the reference data wit= hin the operator that is asynchronously updated at a scheduled frequency us= ing the QueryableState API 3. Output the reference data to Redis, and create an in-memory java cache o= f the reference data within the operator that is asynchronously updated at = a scheduled frequency using Redis API.=20 My understanding is that one of the cons of using Queryable state, is that = if the Flink application that generates the reference data is unavailable, = the Queryable state will not exist - is that correct? If we were to use an asynchronously scheduled 'read' from the distributed c= ache, where should it be done? I was thinking of using ScheduledExecutorSer= vice from within the open method of the Flink operator. What is the best way to get this done? Regards, Hayden Marchant