beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pawel Szczur <pawelszc...@gmail.com>
Subject Helper PTransform MakeKV - would it make sens to add it to SDK?
Date Tue, 05 Jul 2016 14:41:01 GMT
Hi,

Often I encounter a pattern: one gets a big data item, uses some part of it
as key, another as value. Thus I've created simple helper PTransform:
https://gist.github.com/orian/265e5494e61de931bbfa885ac60eb539

This allows me to reuse given function in manu places, sometimes as key
producer and another time as value producer.

The usage is super simple:
PCollection<KV<String, String>> out = input.apply(
MakeKV.of(new SimpleFn(), new Simple2Fn()));

In abstract it takes PCollection<V> and returns PCollection<KV<K,V1>>.,

BTW I had problems with: Cannot provide a Coder for type variable T because
the actual type is unknown due to erasure

I've found:
1.
http://stackoverflow.com/questions/32591914/making-transformations-in-dataflow-generic
2.
http://stackoverflow.com/questions/32591914/making-transformations-in-dataflow-generic
but neither really made it easy to fix (I've been testing the MakeKV with
built in Integer type/coder).

I've found a solution in WithKeys.java in SDK.

Hope it helps someone.

Cheers, Pawel

Mime
View raw message