incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Natkins <na...@cloudera.com>
Subject Re: Looking for some guidance in building a basic Avro pipeline
Date Fri, 07 Dec 2012 01:19:15 GMT
Well, the problem with that is that I really want to work with my objects,
rather than use Avros.generics, because then I'm forced to treat everything
as a GenericData.Record. It's just a pain in the butt.


On Thu, Dec 6, 2012 at 5:17 PM, Josh Wills <josh.wills@gmail.com> wrote:

> You don't want to create an AvroType yourself, you want to call
> o.a.c.types.avro.Avros.records or one of its friends and pass it a Class
> object.
>
> Interesting though, I would still want that case to work correctly.
>
> Josh
> On Dec 6, 2012 5:14 PM, "Jonathan Natkins" <natty@cloudera.com> wrote:
>
>> So I've been futzing with Crunch a bit, and trying to understand how to
>> build a pipeline that outputs Avro data files. Roughly, I'm doing something
>> along these lines:
>>
>>     Schema.Parser schemaParser = new Schema.Parser();
>>     final Schema avroObjSchema = schemaParser.parse(
>> schemaJsonString);
>>
>>     AvroType avroType = new AvroType<MyAvroObject>(MyAvroObject.class,
>>         avroObjSchema, new
>> AvroDeepCopier.AvroReflectDeepCopier<MyAvroObject>(
>>         MyAvroObject.class, avroObjSchema));
>>
>>     PCollection<MyAvroObject> words = logs.parallelDo(new DoFn<String,
>> MyAvroObject>() {
>>       public void process(String line, Emitter<MyAvroObject> emitter) {
>>         emitter.emit(convertStringToAvroObj(line));
>>       }
>>     }, avroType);
>>
>> However, this results in a class cast exception:
>>
>> Exception in thread "main" java.lang.ClassCastException: class
>> com.company.MyAvroObject
>>     at java.lang.Class.asSubclass(Class.java:3039)
>>     at
>> org.apache.crunch.types.writable.Writables.records(Writables.java:250)
>>     at
>> org.apache.crunch.types.writable.WritableTypeFamily.records(WritableTypeFamily.java:86)
>>     at org.apache.crunch.types.PTypeUtils.convert(PTypeUtils.java:61)
>>     at org.apache.crunch.types.writable.WritableTypeFamily.as
>> (WritableTypeFamily.java:135)
>>     at
>> org.apache.crunch.impl.mr.MRPipeline.writeTextFile(MRPipeline.java:319)
>>
>> Anybody have any thoughts? There's got to be a magical incantation that I
>> have slightly off.
>>
>

Mime
View raw message