beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tyler Akidau (JIRA)" <>
Subject [jira] [Commented] (BEAM-1197) Slowly-changing external data as a side input
Date Fri, 14 Apr 2017 21:51:41 GMT


Tyler Akidau commented on BEAM-1197:

A related aspect to consider here is improving the support for temporal joins via side inputs.
[~julianhyde]'s [Streams, joins and temporal tables|]
doc discusses (in a SQL context) what robust semantics here would mean.

> Slowly-changing external data as a side input
> ---------------------------------------------
>                 Key: BEAM-1197
>                 URL:
>             Project: Beam
>          Issue Type: Wish
>          Components: beam-model
>            Reporter: Eugene Kirpichov
> I've seen repeatedly the following pattern: a user wants to join a PCollection against
a slowly-changing external dataset: e.g. a file on GCS, or a Bigtable, etc.
> Side inputs come to mind, but current side input mechanisms don't allow for something
like periodically reloading the side input.
> The best hacky solution I came up with for one use case is documented here:
, we need to do better than this.

This message was sent by Atlassian JIRA

View raw message