beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukasz Cwik <lc...@google.com>
Subject Re: Helper PTransform MakeKV - would it make sens to add it to SDK?
Date Wed, 06 Jul 2016 11:47:59 GMT
To fix your issues with the coder reference, you'll want to apply similar
logic as WithKeys.java
<https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithKeys.java#L114>
which
sets the coder on the output PCollection before returning it to the user.

You'll want to get the output type of the simple function and build a KV
coder from it. The issue is about having a reliable way to get the output
type of the simple function.


On Tue, Jul 5, 2016 at 9:41 AM, Pawel Szczur <pawelszczur@gmail.com> wrote:

> Hi,
>
> Often I encounter a pattern: one gets a big data item, uses some part of
> it as key, another as value. Thus I've created simple helper PTransform:
> https://gist.github.com/orian/265e5494e61de931bbfa885ac60eb539
>
> This allows me to reuse given function in manu places, sometimes as key
> producer and another time as value producer.
>
> The usage is super simple:
> PCollection<KV<String, String>> out = input.apply(
> MakeKV.of(new SimpleFn(), new Simple2Fn()));
>
> In abstract it takes PCollection<V> and returns PCollection<KV<K,V1>>.,
>
> BTW I had problems with: Cannot provide a Coder for type variable T
> because the actual type is unknown due to erasure
>
> I've found:
> 1.
> http://stackoverflow.com/questions/32591914/making-transformations-in-dataflow-generic
> 2.
> http://stackoverflow.com/questions/32591914/making-transformations-in-dataflow-generic
> but neither really made it easy to fix (I've been testing the MakeKV with
> built in Integer type/coder).
>
> I've found a solution in WithKeys.java in SDK.
>
> Hope it helps someone.
>
> Cheers, Pawel
>

Mime
View raw message