beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neville Li (JIRA)" <>
Subject [jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam
Date Mon, 03 Apr 2017 14:41:41 GMT


Neville Li commented on BEAM-302:

We prefer to keep it separate for now mainly for logistics reasons:

- we use SBT with lots of custom logic
- we release very often, once every 1-2 weeks
- we monkey patch Beam bugs, test in our production jobs, before upstream Beam release
- we use a lightweight collaboration model, mainly just Github issues & PRs
- there're only 3 Scio developers at Spotify supporting 150+ internal users and many external
ones, all running on Dataflow

However I also want to point out that nothing should stop those interested from trying it
out or contributing
- we decoupled Dataflow runner as much as possible
- Scio should run on other runners without modification, just a matter of changing dependencies
and arguments
- there're still parts coupled with GCP and Dataflow runner but hopefully we can gradually
decouple them as the file system and other related API improves
- it'd be great to see bug reports and PRs from the community

> Add Scio Scala DSL to Beam
> --------------------------
>                 Key: BEAM-302
>                 URL:
>             Project: Beam
>          Issue Type: Wish
>          Components: sdk-ideas
>            Reporter: Jean-Baptiste Onofré

This message was sent by Atlassian JIRA

View raw message