beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Csaba Kassai <cs...@aliz.ai>
Subject Re: Joining streams
Date Fri, 19 Oct 2018 10:09:26 GMT
Hi Alexander,

I think what you are looking for in the Beam model is the PCollectionList
<https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/values/PCollectionList.html>
and
the Flatten
<https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/values/PCollectionList.html>
transformation. Their usage is described here
<https://beam.apache.org/documentation/programming-guide/#flatten> and here
<https://cloud.google.com/dataflow/model/multiple-pcollections>:

Regards,
Csabi


On Tue, 16 Oct 2018 at 02:52, Kenneth Knowles <kenn@apache.org> wrote:

> Hi Alexander,
>
> In Beam you probably want CoGroupByKey [1]. You might also check out the
> Join Library [2].
>
> Kenn
>
> [1] https://beam.apache.org/documentation/programming-guide/#cogroupbykey
> [2]
> https://beam.apache.org/documentation/sdks/java-extensions/#join-library
>
> On Mon, Oct 15, 2018 at 11:50 AM Alexander Smirnov <
> alexander.smirnoff@gmail.com> wrote:
>
>> Hi everybody,
>>
>> I'm migrating my program from Flink to Beam SDK and stuck on joining two
>> streams.
>>
>> Flink has a dedicated operation "connect" and then you can apply
>> CoFlatMap function to handle messages from different streams.
>>
>> Is there something similar in Beam?
>>
>> Thank you,
>> Alex
>>
>

Mime
View raw message