crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: POJO serialization using generics
Date Wed, 05 Nov 2014 07:13:07 GMT
Hey Kaspar,

I wrote a PType for supporting arbitrarily serializables as part of some
work I was doing on Oryx:

https://github.com/cloudera/oryx/blob/master/computation-common/src/main/java/com/cloudera/oryx/computation/common/types/Serializables.java

I never committed it to Crunch proper b/c it's not the sort of thing we try
to encourage people to do, though like you, I have been in situations where
it was the most expedient solution. ;-)

J

On Tue, Nov 4, 2014 at 10:50 PM, Kaspar Fischer <kaspar.fischer@dreizak.com>
wrote:

> Hi everybody,
>
> I am using Crunch with Avros serialization. Some intermediate objects of
> my pipeline are POJO's that have fields with generics. When running the
> pipeline I get:
>
> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown
> type: T
>
> at
> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:220)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
>
> at
> org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
>
> at
> org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
>
> at
> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:197)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
>
> at
> org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
>
> at
> org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
>
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154)
>
> at org.apache.crunch.types.avro.Avros.reflects(Avros.java:275)
>
> at org.apache.crunch.types.avro.Avros.containers(Avros.java:264)
>
> at org.apache.crunch.types.avro.Avros.records(Avros.java:252)
>
> How can I get around this? Is there a way to serialize the object using
> Java serialization (or Kryo)?
>
> Thanks for any pointers,
> Kaspar
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message