crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: Scrunch: Using Avro-generated Java enums in compound types
Date Tue, 30 Sep 2014 16:00:34 GMT
I think that adding the Java enum support directly is the way to go; I'm
trying to avoid hacking Avro reflection where I can. Here's the patch,
building off of the enum PType support in Java Crunch:

https://issues.apache.org/jira/browse/CRUNCH-472

On Tue, Sep 30, 2014 at 5:14 AM, David Whiting <davidwhiting@gmail.com>
wrote:

> We have encountered a strange problem in our Scrunch code when attempting
> to serialize Java enum types (as generated by Avro).
>
> Basically, if you create an Avro-schema with an enum-typed field, then a
> Java enum class will be generated for that field. When you create a Scrunch
> pipeline to use it, and use it within a compound type as an intermediate
> value, it fails when spilling to disk because the ReflectDatumWriter cannot
> instantiate the enum type.
>
> Inspecting the implicit PTypeH parameter passed to the offending function
> (flatMap to a 4-tuple in this case), we see that it resolves to
> quads(records[MyEnumType), strings, strings, strings). The records gets
> implemented by the PTypeFamiliy (AvroTypeFamily in this case), which
> delegates to containers and then reflects, which in tern delegates to the
> Avro standard reflection stuff. I would expect this to have no problem with
> an enum type, but for some reason it is trying to instantiate it instead of
> using it as an enum.
>
> Is there some special case for Java enums missing in PTypeH, or have I
> maybe done something else wrong somewhere?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message