beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismaël Mejía <ieme...@gmail.com>
Subject Re: Streaming pipelines in all SDKs!
Date Fri, 14 Jun 2019 20:38:56 GMT
Thanks

On Thu, Jun 13, 2019 at 2:02 PM Łukasz Gajowy <lgajowy@apache.org> wrote:
>
> Created a PR: https://github.com/apache/beam/pull/8846
>
> śr., 12 cze 2019 o 11:40 Ismaël Mejía <iemejia@gmail.com> napisał(a):
>>
>> Can you please add this to the design documents webpage.
>> https://beam.apache.org/contribute/design-documents/
>>
>> On Fri, May 10, 2019 at 11:50 AM Maximilian Michels <mxm@apache.org> wrote:
>> >
>> > > So, FlinkRunner has some sort of special support for executing UnboundedSource
via the runner in the portable world ? I see a transform override for bounded sources in PortableRunner
[1] but nothing for unbounded sources.
>> >
>> > It's in the translation code:
>> > https://github.com/apache/beam/blob/6679b00138a5b82a6a55e7bc94c453957cea501c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java#L216
>> >
>> > For migration I think that's a valid path, especially because Runners
>> > already have the translation code in place. We can later swap-out the
>> > UnboundedSource translation with the SDF wrapper.
>> >
>> > -Max
>> >
>> > On 09.05.19 22:46, Robert Bradshaw wrote:
>> > > From: Chamikara Jayalath <chamikara@google.com>
>> > > Date: Thu, May 9, 2019 at 7:49 PM
>> > > To: dev
>> > >
>> > >> From: Maximilian Michels <mxm@apache.org>
>> > >> Date: Thu, May 9, 2019 at 9:21 AM
>> > >> To: <dev@beam.apache.org>
>> > >>
>> > >>> Thanks for sharing your ideas for load testing!
>> > >>>
>> > >>>> According to other contributors knowledge/experience: I noticed
that streaming with KafkaIO is currently supported by wrapping the ExternalTransform in Python
SDK. Do you think that streaming pipelines will "just work" with the current state of portability
if I do the same for UnboundedSyntheticSource or is there something else missing?
>> > >>>
>> > >>> Basically yes, but it requires a bit more effort than just wrapping
>> > >>> about ExternalTransform. You need to provide an ExternalTransformBuilder
>> > >>> for the transform to be configured externally.
>> > >>>
>> > >>> In portability UnboundedSources can only be supported via SDF.
To still
>> > >>> be able to use legacy IO which uses UnboundedSource the Runner
has to
>> > >>> supply this capability (which the Flink Runner supports). This
will
>> > >>> likely go away if we have an UnboundedSource SDF Wrapper :)
>> > >>
>> > >>
>> > >> So, FlinkRunner has some sort of special support for executing UnboundedSource
via the runner in the portable world ? I see a transform override for bounded sources in PortableRunner
[1] but nothing for unbounded sources.
>> > >>
>> > >> Agree, that we cannot properly support cross-language unbounded sources
till we have SDF and a proper unbounded source to SDF wrapper.
>> > >
>> > > That is correct. Go will need SDF support as well.
>> > >
>> > > As waiting on implementing the expansion service, except for the
>> > > vending of extra artifacts (which will be an extension), we discussed
>> > > this earlier and it's considered stable and ready to build on now.
>> > >

Mime
View raw message