Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Fri, 28 Apr 2017 15:10:04 +0000 (UTC)
From: "Aljoscha Krettek (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13049004.1488915701000.79457.1493392204890@Atlassian.JIRA>
In-Reply-To: <JIRA.13049004.1488915701000@Atlassian.JIRA>
References: <JIRA.13049004.1488915701000@Atlassian.JIRA> <JIRA.13049004.1488915701098@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-1641) Support synchronized processing time
 in Flink runner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 28 Apr 2017 15:10:08 -0000


    [ https://issues.apache.org/jira/browse/BEAM-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988973#comment-15988973 ] 

Aljoscha Krettek commented on BEAM-1641:
----------------------------------------

Quick side note: In Flink, if you set your "stream time characteristic" (a bit of a mouthful) to _ingestion time_ then you will essentially get the behaviour of synchronised processing time. What this does is assign the current system time as the element timestamp at the sources and generates watermarks based on the current system time. After the sources it uses the same mechanism as event-time. The problem is that it uses exactly the same mechanism, i.e. you can either have proper event-time or ingestion time and there is actually no difference to operators. All they see is "event time" and it might be actual event time or ingestion time.

I think the proper solution would be for Flink to track ingestion time (which is synchronised processing time) and event-time at the same time in one job.

[~kenn] and [~lzljs3620320], what do you think?

> Support synchronized processing time in Flink runner
> ----------------------------------------------------
>
>                 Key: BEAM-1641
>                 URL: https://issues.apache.org/jira/browse/BEAM-1641
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>            Reporter: Kenneth Knowles
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>             Fix For: First stable release
>
>
> The "continuation trigger" for a processing time trigger is a synchronized processing time trigger. Today, this throws an exception in the FlinkRunner.
> The supports the following:
>  - GBK1
>  - GBK2
> When GBK1 fires due to processing time past the first element in the pane and that element arrives at GBK2, it will wait until all the other upstream keys have also processed and emitted corresponding data.
> Sorry for the terseness of explanation - writing quickly so I don't forget.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)