beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jingsong Lee (JIRA)" <>
Subject [jira] [Commented] (BEAM-1641) Support synchronized processing time in Flink runner
Date Tue, 02 May 2017 02:51:04 GMT


Jingsong Lee commented on BEAM-1641:

There are some differences between the processing of event time and synchronised processing
time in {{DirectRunner}}. The Source just emit the {{BoundedWindow.TIMESTAMP_MAX_VALUE}} as
the synchronizedProcessingTime, and the downStream use {{min(, synchronizedProcessingInputWatermark.get())}}
to generate synchronizedProcessingTime.
But I think from the fundamental point of view, ingestion time and synchronized processing
time have produced almost the same effect. So I think we can use ingestion time and let Flink
track ingestion and event time at the same time.

> Support synchronized processing time in Flink runner
> ----------------------------------------------------
>                 Key: BEAM-1641
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>            Reporter: Kenneth Knowles
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>             Fix For: First stable release
> The "continuation trigger" for a processing time trigger is a synchronized processing
time trigger. Today, this throws an exception in the FlinkRunner.
> The supports the following:
>  - GBK1
>  - GBK2
> When GBK1 fires due to processing time past the first element in the pane and that element
arrives at GBK2, it will wait until all the other upstream keys have also processed and emitted
corresponding data.
> Sorry for the terseness of explanation - writing quickly so I don't forget.

This message was sent by Atlassian JIRA

View raw message