crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barry,Nathan" <NBA...@CERNER.COM>
Subject scrunch exception mapping Avros to HBase Puts
Date Wed, 19 Mar 2014 01:12:53 GMT
I’m reworking some existing code migrating the logic [1] to leverage scrunch going away from
crunch based logic [2] and I’m getting exceptions when trying to map a scrunch PCollection
of avros to a PCollection of HBase Puts – this happens when using scala-2.9.3 + scrunch
0.8.0 as well as scala-2.10.3 + scrunch 0.8.2+32, though the details in the stack are slightly
different.  The same logic using crunch in scala does not have the same problem

In both cases the exception is stems from SpecificData.getSchema, but from there the 2.9/0.8
stack [3] differs slightly from the 2.10/0.8.2 stack.

Has anyone seen anything like this before?  Or know of ways to potentially work around it?
 This is not a show stopper but the scrunch code is so much simpler I’d prefer to go that
route if at all possible.


Thanks in advance, Nathan

[1] concepts.map(concept => conn.getSession().newPut(tableName, entityKey(concept.getId))).setColumn(FAMILY,
COLUMN, concept).toRawPut)

[2] concepts.parallelDo("convertMyAvroToPut", new ConvertToPutFn[MyAvro](timestamp, tableName,

        concept => entityKey(concept.getId), (putRequest, concept)  => putRequest.setColumn(FAMILY,
COLUMN, concept).toRawPut), org.apache.crunch.types.writable.Writables.writables(classOf[Put]))

[3] org.apache.avro.AvroTypeException: Map key class not String: byte[]
at org.apache.crunch.scrunch.ScalaSafeReflectData.createSchema(ScalaSafeReflectData.java:105)
at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
at org.apache.crunch.scrunch.ScalaSafeReflectData.createSchema(ScalaSafeReflectData.java:170)
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154)
at org.apache.crunch.types.avro.Avros.reflects(Avros.java:239)
at org.apache.crunch.types.avro.Avros.containers(Avros.java:229)
at org.apache.crunch.types.avro.Avros.records(Avros.java:217)
at org.apache.crunch.types.avro.AvroTypeFamily.records(AvroTypeFamily.java:92)
at org.apache.crunch.scrunch.PTypeFamily$class.records(PTypeFamily.scala:40)
at org.apache.crunch.scrunch.Avros$.records(PTypeFamily.scala:122)
at org.apache.crunch.scrunch.PTypeH$$anon$15.get(Conversions.scala:135)
at org.apache.crunch.scrunch.PCollection.map(PCollection.scala:36)


[4] java.lang.ClassCastException: sun.reflect.generics.reflectiveObjects.GenericArrayTypeImpl
cannot be cast to java.lang.Class
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:273)
at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154)
at org.apache.crunch.types.avro.Avros.reflects(Avros.java:279)
at org.apache.crunch.types.avro.Avros.containers(Avros.java:269)
at org.apache.crunch.types.avro.Avros.records(Avros.java:257)
at org.apache.crunch.types.avro.AvroTypeFamily.records(AvroTypeFamily.java:93)
at org.apache.crunch.scrunch.PTypeFamily$class.records(PTypeFamily.scala:47)
at org.apache.crunch.scrunch.Avros$.records(PTypeFamily.scala:129)
at org.apache.crunch.scrunch.PTypeH$$anon$15.get(Conversions.scala:135)
at org.apache.crunch.scrunch.PCollection.map(PCollection.scala:36)



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation
and are intended only for the addressee. The information contained in this message is confidential
and may constitute inside or non-public information under international, federal, or state
securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such
information is strictly prohibited and may be unlawful. If you are not the addressee, please
promptly delete this message and notify the sender of the delivery error by e-mail or you
may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

Mime
View raw message