From user-return-23384-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Oct 2 23:31:48 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 044F5180638 for ; Tue, 2 Oct 2018 23:31:47 +0200 (CEST) Received: (qmail 31645 invoked by uid 500); 2 Oct 2018 21:31:46 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 31635 invoked by uid 99); 2 Oct 2018 21:31:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Oct 2018 21:31:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 52B5E183E0C for ; Tue, 2 Oct 2018 21:31:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.485 X-Spam-Level: *** X-Spam-Status: No, score=3.485 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, NML_ADSP_CUSTOM_MED=1.2, SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.972, URI_HEX=1.313] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id zXo6DfDTTrpt for ; Tue, 2 Oct 2018 21:31:45 +0000 (UTC) Received: from n4.nabble.com (n4.nabble.com [199.38.86.66]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 8B5015F2FE for ; Tue, 2 Oct 2018 21:31:44 +0000 (UTC) Received: from n4.nabble.com (localhost [127.0.0.1]) by n4.nabble.com (Postfix) with ESMTP id 9FDDE1FE2318 for ; Tue, 2 Oct 2018 16:31:43 -0500 (CDT) Date: Tue, 2 Oct 2018 16:31:43 -0500 (CDT) From: Anil To: user@flink.apache.org Message-ID: <1538515903584-0.post@n4.nabble.com> Subject: Flink Checkpoint Barrier in case of Join MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'm trying to understand when will Flink's Stream Barrier (for checkpoint) = be emitted by the join operator.=20 Consider a query like - select * from stream_1 a1 INNER JOIN stream_2 a2 on a2.orderId =3D a1.orderId group by HOP(a1.proctime, INTERVAL '1' HOUR, INTERVAL '1' DAY), a1.restaurantId Since I'm using a Hopping window on 1 day here, Flink will have to cache my entire 1 day events.=20 The join operator will receive stream barrier from the previous operator.= =20 Join operator will emit one stream barrier but I'm not sure on what basis and when will it be emitted.=20 Any help will be appreciated. Thanks! From Flink's documentation -=20 ```A core element in Flink=E2=80=99s distributed snapshotting are the strea= m barriers. These barriers are injected into the data stream and flow with th= e records as part of the data stream. The point where the barriers for snapshot n are injected (let=E2=80=99s call it Sn) is the position in the s= ource stream up to which the snapshot covers the data. The barriers then flow downstream. When an intermediate operator has received a barrier for snapshot n from all of its input streams, it emits a barrier for snapshot n into all of its outgoing streams. Once a sink operator (the end of a streaming DAG) has received the barrier n from all of its input streams, it acknowledges that snapshot n to the checkpoint coordinator. After all sinks have acknowledged a snapshot, it is considered completed.``` -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.= com/