beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Pouttu-Clarke (JIRA)" <>
Subject [jira] [Commented] (BEAM-91) Retractions
Date Thu, 23 Jun 2016 00:35:16 GMT


Matt Pouttu-Clarke commented on BEAM-91:

Yes agreed it is not clear yet from the docs how this relates directly to Beam.  However this
is mainly a terminology issue in my perspective.  The bespoke systems I have built over the
last few years to unify batch and stream processing all rely on data versioning to ensure
point-in-session consistency (watermarks) across streams and all data derived from streams
such as aggregates, transforms, splits, and replicas.  

There is no hard dependency on a configuration service but it is critical to keep a current
water mark and all historical watermarks in a system of record.  This could be as simple as
a shared file system or as complex as etcd.  

That aside the versioning model I set forward for flatbuffers is an example using more recent
technologies. I have done the same with relational tables and Avro in the past.

I'll work on the examples of how the versioning model feeds aggregate refresh and hopefully
it will become more clear.  

> Retractions
> -----------
>                 Key: BEAM-91
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Tyler Akidau
>            Assignee: Frances Perry
>   Original Estimate: 672h
>  Remaining Estimate: 672h
> We still haven't added retractions to Beam, even though they're a core part of the model.
We should document all the necessary aspects (uncombine, reverting DoFn output with DoOvers,
sink integration, source-level retractions, etc), and then implement them.

This message was sent by Atlassian JIRA

View raw message