crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sankash Shankar <sank...@wealthfront.com>
Subject How to write a generic transform method that will act upon generated avro objects in a generic fashion
Date Mon, 22 Jun 2015 20:17:31 GMT
Hello.

I am writing a Crunch job that takes in an arbitrary class that extends
SpecificRecord and performs a transformation on the fields in the class. I
am attempting to write a parallelDo function on these classes, but

public static PCollection<String> function(PCollection<? extends
SpecificRecord> coll) {
  coll.parallelDo(new DoFn<? extends SpecificRecord, String>() {
    ...
  }, Avros.strings());
}

will not compile given it expects a type at compile-time

*will not compile given it expects a type at compile time, while *

public static PCollection<String>
transformAvroToCsv(PCollection<SpecificRecord> coll) {
  coll.parallelDo(new DoFn<SpecificRecord, String>() {
    @Override
    public void process(SpecificRecord input, Emitter<String> emitter) {
    }
  }, Avros.strings());
  return null;
}

*will fail at run-time due to SpecificRecord not having an init
constructor.*

What is the standard way for taking in generic avro records and having a
generic
transform method to call on them?

Thanks.

Mime
View raw message