beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Sisk (JIRA)" <>
Subject [jira] [Commented] (BEAM-1025) User guide - "How to create Beam IO Transforms"
Date Wed, 01 Mar 2017 22:15:45 GMT


Stephen Sisk commented on BEAM-1025:

thanks JB! 

More thinking as I've been looking into what we already have:  Pipeline IO probably has enough
content associated with it that we're going to want it to have it's own page (or set of pages.)

I'm going to move in that direction, and try to reconcile the python documentation as we go.

cc [~melap] and [~chamikara] who I believe are also interested in this.

> User guide - "How to create Beam IO Transforms"
> -----------------------------------------------
>                 Key: BEAM-1025
>                 URL:
>             Project: Beam
>          Issue Type: Task
>          Components: website
>            Reporter: Stephen Sisk
>            Assignee: Stephen Sisk
> Beam has javadocs for how to create a read or write transform, but no friendly user guide
on how to get started using BoundedSource/BoundedReader.
> This should cover:
> * background on beam's source/sink API design 
> * design patterns
> * evaluating different data sources (eg, what are the properties of a pub sub system
that affect how you should write your UnboundedSource? What is the best design for reading
from a NoSql style source?)
> * testing - how to write unit, integration (and once we have them, performance tests)
> * public API recommendations
> This is related, but not strictly overlapping with: 
> - the Dataflow SDK documentation for "Custom Sources and Sinks"  contains some info about
writing Sources/Sinks, but it is somewhat out of date, and doesn't reflect the things we've
learned recently.

This message was sent by Atlassian JIRA

View raw message