beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Pouttu-Clarke (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-91) Retractions
Date Wed, 22 Jun 2016 00:34:57 GMT

    [ https://issues.apache.org/jira/browse/BEAM-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343104#comment-15343104
] 

Matt Pouttu-Clarke commented on BEAM-91:
----------------------------------------

https://github.com/LamdaFu/bloklinx/wiki
^^ provides the basic description
https://github.com/LamdaFu/bloklinx/wiki/Bloklinx-Schema-(flatbuffers)
^^ provides definition of the basic mapping to flatbuffer serialization

I am working on examples of basic versioning as well as branching / merge / change data processing

The question I get asked most is why an "UPDATE" is composed of a redaction followed by an
assertion rather than just one record.  The answer is that this provides several huge benefits
including very efficient refresh of downstream aggregations, splits, merges, easier data diff,
and much easier reconciliation.  This will become clear with my subsequent examples to come.

> Retractions
> -----------
>
>                 Key: BEAM-91
>                 URL: https://issues.apache.org/jira/browse/BEAM-91
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Tyler Akidau
>            Assignee: Frances Perry
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> We still haven't added retractions to Beam, even though they're a core part of the model.
We should document all the necessary aspects (uncombine, reverting DoFn output with DoOvers,
sink integration, source-level retractions, etc), and then implement them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message