beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chamikara Jayalath <chamik...@google.com>
Subject Re: Using Beam Built-in I/O Transforms with an external framework.
Date Wed, 18 Sep 2019 21:17:29 GMT
Hi Pulasthi,

This might be possible but I don't know if anybody has done this. API of
Beam sources are no different from other Beam PTransforms and we highly
recommend hiding away various implementations of source framework related
abstractions in a composite transform [1]. So what you are looking for is a
way to use a Beam transform in a separate system.

Thanks,
Cham

[1]
https://beam.apache.org/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else

On Wed, Sep 18, 2019 at 1:52 PM Pulasthi Supun Wickramasinghe <
pulasthi911@gmail.com> wrote:

> Hi Dev's
>
> We have a big data processing framework named Twister2, and wanted to know
> if there is any way we could leverage the I/O Transforms that are built
> into Apache Beam externally. That is rather than using it in a Beam
> pipeline just use them as data sources in our project. Just wanted to check
> with the dev's if such an approach has been done by anyone before, so we
> could get some pointers on how this could be done. Any pointers in the
> right direction would be highly appreciated.
>
> Best Regards,
> Pulasthi
>
> --
> Pulasthi S. Wickramasinghe
> PhD Candidate  | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035 <(224)%20386-9035>
>

Mime
View raw message