From issues-return-168116-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed May 23 11:18:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id AE9FD180679 for ; Wed, 23 May 2018 11:18:03 +0200 (CEST) Received: (qmail 29526 invoked by uid 500); 23 May 2018 09:18:02 -0000 Mailing-List: contact issues-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list issues@flink.apache.org Received: (qmail 29517 invoked by uid 99); 23 May 2018 09:18:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 May 2018 09:18:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 4E69FC006A for ; Wed, 23 May 2018 09:18:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id J1XwboZA6rm5 for ; Wed, 23 May 2018 09:18:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id E9B395F3BD for ; Wed, 23 May 2018 09:18:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 82AD9E0047 for ; Wed, 23 May 2018 09:18:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 46B1D21094 for ; Wed, 23 May 2018 09:18:00 +0000 (UTC) Date: Wed, 23 May 2018 09:18:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@flink.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FLINK-6968) Store streaming, updating tables with unique key in queryable state MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/FLINK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16486957#comment-16486957 ] ASF GitHub Bot commented on FLINK-6968: --------------------------------------- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/5688#discussion_r190176842 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/KeyedProcessFunctionWithCleanupState.scala --- @@ -44,8 +45,20 @@ abstract class KeyedProcessFunctionWithCleanupState[K, I, O](queryConfig: Stream protected def registerProcessingCleanupTimer( ctx: KeyedProcessFunction[K, I, O]#Context, currentTime: Long): Unit = { - if (stateCleaningEnabled) { + registerCleanupTimer(ctx, currentTime, TimeDomain.PROCESSING_TIME) + } + protected def registerEventCleanupTimer( --- End diff -- We implemented state cleanup as processing time because it is easier to reason about for users and doesn't interfere that much with event-time processing (it is not possible to distinguish timers yet). However, it also has a few short comings such as cleared state when recovering a query from a savepoint (which we don't really encourage at the moment). Anyway, introducing event-time state cleanup should definitely go into a separate issue and PR. > Store streaming, updating tables with unique key in queryable state > ------------------------------------------------------------------- > > Key: FLINK-6968 > URL: https://issues.apache.org/jira/browse/FLINK-6968 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL > Reporter: Fabian Hueske > Assignee: Renjie Liu > Priority: Major > > Streaming tables with unique key are continuously updated. For example queries with a non-windowed aggregation generate such tables. Commonly, such updating tables are emitted via an upsert table sink to an external datastore (k-v store, database) to make it accessible to applications. > This issue is about adding a feature to store and maintain such a table as queryable state in Flink. By storing the table in Flnk's queryable state, we do not need an external data store to access the results of the query but can query the results directly from Flink. -- This message was sent by Atlassian JIRA (v7.6.3#76005)