beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukasz Cwik <>
Subject Re: Helper PTransform MakeKV - would it make sens to add it to SDK?
Date Wed, 06 Jul 2016 11:47:59 GMT
To fix your issues with the coder reference, you'll want to apply similar
logic as
sets the coder on the output PCollection before returning it to the user.

You'll want to get the output type of the simple function and build a KV
coder from it. The issue is about having a reliable way to get the output
type of the simple function.

On Tue, Jul 5, 2016 at 9:41 AM, Pawel Szczur <> wrote:

> Hi,
> Often I encounter a pattern: one gets a big data item, uses some part of
> it as key, another as value. Thus I've created simple helper PTransform:
> This allows me to reuse given function in manu places, sometimes as key
> producer and another time as value producer.
> The usage is super simple:
> PCollection<KV<String, String>> out = input.apply(
> MakeKV.of(new SimpleFn(), new Simple2Fn()));
> In abstract it takes PCollection<V> and returns PCollection<KV<K,V1>>.,
> BTW I had problems with: Cannot provide a Coder for type variable T
> because the actual type is unknown due to erasure
> I've found:
> 1.
> 2.
> but neither really made it easy to fix (I've been testing the MakeKV with
> built in Integer type/coder).
> I've found a solution in in SDK.
> Hope it helps someone.
> Cheers, Pawel

View raw message