beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Commented] (BEAM-71) Watermark library
Date Tue, 16 Jan 2018 20:20:00 GMT


Eugene Kirpichov commented on BEAM-71:

As far as SDF is concerned, the only relevant method is DoFn.ProcessContext.updateWatermark()

This issue should probably be implemented as a utility class that people can use in arbitrary
ways, both from an UnboundedSource and from an SDF. E.g. an UnboundedReader.getWatermark()
could delegate to utility.getWatermark(), and an SDF could call c.updateWatermark(utility.getWatermark()).

Here's another interesting example of computing watermarks (in an SDF): [] -
search for "watermark", it's pretty fancy, though it might be too fancy to consider it as
something that the current issue should cover.

> Watermark library
> -----------------
>                 Key: BEAM-71
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Assignee: Tarush Grover
>            Priority: Major
>              Labels: Watermark
> See also: BEAM-70
> We should have a library of watermark implementations:
> * now: for when items use arrival timestamps
> * latest element: for when events arrive in timestamp order
> * moving average over last K time intervals
> * bucketed moving average, ...

This message was sent by Atlassian JIRA

View raw message