Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2FECD200C10 for ; Fri, 20 Jan 2017 06:00:39 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 2E7DA160B57; Fri, 20 Jan 2017 05:00:39 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 79896160B54 for ; Fri, 20 Jan 2017 06:00:38 +0100 (CET) Received: (qmail 67666 invoked by uid 500); 20 Jan 2017 05:00:37 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 67657 invoked by uid 99); 20 Jan 2017 05:00:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Jan 2017 05:00:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 470041A0362 for ; Fri, 20 Jan 2017 05:00:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.999 X-Spam-Level: X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id hMXTLjeVCZJe for ; Fri, 20 Jan 2017 05:00:32 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id EA4A45FB1E for ; Fri, 20 Jan 2017 05:00:31 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 1A8B5E008E for ; Fri, 20 Jan 2017 05:00:31 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id BC64825285 for ; Fri, 20 Jan 2017 05:00:30 +0000 (UTC) Date: Fri, 20 Jan 2017 05:00:30 +0000 (UTC) From: "Kenneth Knowles (JIRA)" To: commits@beam.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (BEAM-241) Not easy for runners to get late-data dropping MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 20 Jan 2017 05:00:39 -0000 [ https://issues.apache.org/jira/browse/BEAM-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831187#comment-15831187 ] Kenneth Knowles commented on BEAM-241: -------------------------------------- At this point is it very easily {{DoFnRunners.lateDataDroppingRunner(underlyingDoFnRunner, ...)}} > Not easy for runners to get late-data dropping > ---------------------------------------------- > > Key: BEAM-241 > URL: https://issues.apache.org/jira/browse/BEAM-241 > Project: Beam > Issue Type: Bug > Components: runner-core > Reporter: Mark Shields > Assignee: Kenneth Knowles > > Quite by accident realized the Flink runner delegates to GroupAlsoByWindowViaWindowSetDoFn for GBK, which in turn delegates to ReduceFnRunner. The latter now assumes no messages will arrive beyond the 'garbage collection' time of their target window(s). > The Dataflow runner interposes a LateDataDroppingDoFnRunner into the path so as to drop those too-late messages. That's done (I think) using DoFnRunners.createDefault. > I don't think the Flink runner does that. > We need a nice runner-friendly way of dealing with the too-late data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)