beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <>
Subject [jira] [Commented] (BEAM-1641) Support synchronized processing time in Flink runner
Date Fri, 28 Apr 2017 15:10:04 GMT


Aljoscha Krettek commented on BEAM-1641:

Quick side note: In Flink, if you set your "stream time characteristic" (a bit of a mouthful)
to _ingestion time_ then you will essentially get the behaviour of synchronised processing
time. What this does is assign the current system time as the element timestamp at the sources
and generates watermarks based on the current system time. After the sources it uses the same
mechanism as event-time. The problem is that it uses exactly the same mechanism, i.e. you
can either have proper event-time or ingestion time and there is actually no difference to
operators. All they see is "event time" and it might be actual event time or ingestion time.

I think the proper solution would be for Flink to track ingestion time (which is synchronised
processing time) and event-time at the same time in one job.

[~kenn] and [~lzljs3620320], what do you think?

> Support synchronized processing time in Flink runner
> ----------------------------------------------------
>                 Key: BEAM-1641
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>            Reporter: Kenneth Knowles
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>             Fix For: First stable release
> The "continuation trigger" for a processing time trigger is a synchronized processing
time trigger. Today, this throws an exception in the FlinkRunner.
> The supports the following:
>  - GBK1
>  - GBK2
> When GBK1 fires due to processing time past the first element in the pane and that element
arrives at GBK2, it will wait until all the other upstream keys have also processed and emitted
corresponding data.
> Sorry for the terseness of explanation - writing quickly so I don't forget.

This message was sent by Atlassian JIRA

View raw message