crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sankash Shankar <>
Subject How to write a generic transform method that will act upon generated avro objects in a generic fashion
Date Mon, 22 Jun 2015 20:17:31 GMT

I am writing a Crunch job that takes in an arbitrary class that extends
SpecificRecord and performs a transformation on the fields in the class. I
am attempting to write a parallelDo function on these classes, but

public static PCollection<String> function(PCollection<? extends
SpecificRecord> coll) {
  coll.parallelDo(new DoFn<? extends SpecificRecord, String>() {
  }, Avros.strings());

will not compile given it expects a type at compile-time

*will not compile given it expects a type at compile time, while *

public static PCollection<String>
transformAvroToCsv(PCollection<SpecificRecord> coll) {
  coll.parallelDo(new DoFn<SpecificRecord, String>() {
    public void process(SpecificRecord input, Emitter<String> emitter) {
  }, Avros.strings());
  return null;

*will fail at run-time due to SpecificRecord not having an init

What is the standard way for taking in generic avro records and having a
transform method to call on them?


View raw message