beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Pouttu-Clarke (JIRA)" <>
Subject [jira] [Commented] (BEAM-91) Retractions
Date Tue, 21 Jun 2016 15:36:57 GMT


Matt Pouttu-Clarke commented on BEAM-91:

I've solved this problem successfully several times since around 2007.  It requires implementing
data versioning and treating data much like you would treat code in github.  You could also
call it streaming with parallel universe support, as some consumers may not want or need your
redactions, while others may have critical need of them (much like in the source code world
some users do not want immediate "upgrades").  Also, please note that it is just as important
to support redacting structural changes as it is to support redacting data changes.  I have
mature and battle tested designs in this area if there's interest.

> Retractions
> -----------
>                 Key: BEAM-91
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Tyler Akidau
>            Assignee: Frances Perry
>   Original Estimate: 672h
>  Remaining Estimate: 672h
> We still haven't added retractions to Beam, even though they're a core part of the model.
We should document all the necessary aspects (uncombine, reverting DoFn output with DoOvers,
sink integration, source-level retractions, etc), and then implement them.

This message was sent by Atlassian JIRA

View raw message