beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henning Rohde <hero...@google.com>
Subject Re: Reading all elements of a PCollection after running beam go pipeline
Date Sat, 05 May 2018 00:14:35 GMT
Great!

On Fri, May 4, 2018 at 4:37 PM 8 Gianfortoni <8@tokentransit.com> wrote:

> Thanks for the workaround! That should work for me.
>
> On Fri, May 4, 2018, 1:51 PM Henning Rohde <herohde@google.com> wrote:
>
>> Hey there,
>>
>>  Until side input is fully supported, you can use GBK with a fixed key to
>> get all elements in a single bundle (assuming global windowing). That is
>> how textio.Write works internally to produce a single file currently:
>>
>> https://github.com/apache/beam/blob/90ba47114997e75a3a7daa1e8db92768d05e5432/sdks/go/pkg/beam/io/textio/textio.go#L144
>>
>>
>>
>> Thanks,
>>  Henning
>>
>>
>> On Fri, May 4, 2018 at 11:41 AM 8 Gianfortoni <8@tokentransit.com> wrote:
>>
>>> Hi dev team,
>>>
>>> I would like to be able to read the entire results of a PCollection
>>> serially after running beam. In other frameworks this is fairly
>>> straightforward, but I don't understand how one might do this with the Beam
>>> Go SDK.
>>>
>>> I guess I can read in a file that I write, but I want to be able to read
>>> the elements in struct format. I looked through a bunch of examples but
>>> don't see any that do this.
>>>
>>> Thanks,
>>> 8
>>>
>>

Mime
View raw message