crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kaspar Fischer <kaspar.fisc...@dreizak.com>
Subject Re: POJO serialization using generics
Date Wed, 05 Nov 2014 07:28:48 GMT
Hey Josh,

Great, very useful!

Thanks a lot for sharing this. 

Cheers,
Kaspar 



> On 04.11.2014, at 23:13, Josh Wills <jwills@cloudera.com> wrote:
> 
> Hey Kaspar,
> 
> I wrote a PType for supporting arbitrarily serializables as part of some work I was doing
on Oryx:
> 
> https://github.com/cloudera/oryx/blob/master/computation-common/src/main/java/com/cloudera/oryx/computation/common/types/Serializables.java
> 
> I never committed it to Crunch proper b/c it's not the sort of thing we try to encourage
people to do, though like you, I have been in situations where it was the most expedient solution.
;-)
> 
> J
> 
>> On Tue, Nov 4, 2014 at 10:50 PM, Kaspar Fischer <kaspar.fischer@dreizak.com>
wrote:
>> Hi everybody,
>> 
>> I am using Crunch with Avros serialization. Some intermediate objects of my pipeline
are POJO's that have fields with generics. When running the pipeline I get:
>> 
>> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown type: T
>> 	at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:220)
>> 	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
>> 	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
>> 	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
>> 	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
>> 	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
>> 	at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:197)
>> 	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
>> 	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
>> 	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
>> 	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
>> 	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
>> 	at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154)
>> 	at org.apache.crunch.types.avro.Avros.reflects(Avros.java:275)
>> 	at org.apache.crunch.types.avro.Avros.containers(Avros.java:264)
>> 	at org.apache.crunch.types.avro.Avros.records(Avros.java:252)
>> 
>> How can I get around this? Is there a way to serialize the object using Java serialization
(or Kryo)?
>> 
>> Thanks for any pointers,
>> Kaspar
> 
> 
> 
> -- 
> Director of Data Science
> Cloudera
> Twitter: @josh_wills

Mime
View raw message