beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pawel Szczur <>
Subject Helper PTransform MakeKV - would it make sens to add it to SDK?
Date Tue, 05 Jul 2016 14:41:01 GMT

Often I encounter a pattern: one gets a big data item, uses some part of it
as key, another as value. Thus I've created simple helper PTransform:

This allows me to reuse given function in manu places, sometimes as key
producer and another time as value producer.

The usage is super simple:
PCollection<KV<String, String>> out = input.apply(
MakeKV.of(new SimpleFn(), new Simple2Fn()));

In abstract it takes PCollection<V> and returns PCollection<KV<K,V1>>.,

BTW I had problems with: Cannot provide a Coder for type variable T because
the actual type is unknown due to erasure

I've found:
but neither really made it easy to fix (I've been testing the MakeKV with
built in Integer type/coder).

I've found a solution in in SDK.

Hope it helps someone.

Cheers, Pawel

View raw message