beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chad Dombrova <>
Subject Re: DataflowRunner | Cross-language
Date Mon, 08 Jun 2020 19:57:38 GMT
Hi all,
quick followup question:

> small correction. While the new runner will be available with Beam 2.21,
>> the Cross-Language support will be available in 2.22.
>> There will be limitations in the initial set of connectors you can use
>> with Cross-Lang. But at least you will have something to test with,
>> starting in 2.22
> To clarify, we're not actually prohibiting any other
> cross-langauge transforms being used, but Kafka is the only one that'll be
> extensively tested and supported at this time.

We're currently using the Flink runner with external Java PubSubIO
transforms in our python pipelines because there is no pure python option.
 In its non-portable past, Dataflow has had its own native implementation
of PubSubIO, that got switched out at runtime, so there was no need to use
external transforms there.  What's the story around PubSubIO when using
Dataflow + portability?  If we were to switch from Flink to Dataflow, would
we continue to use external Java PubSubIO transforms, or is there still
some special treatment of pubsub for Portable Dataflow?


View raw message