avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <scottca...@apache.org>
Subject Re: Issue writing union in avro?
Date Sat, 06 Apr 2013 20:36:17 GMT
This is due to using the JSON encoding for avro and not the binary encoding.
It would appear that the Python version is a little bit lax on the spec.
Some have built variations of the JSON encoding that do not label the union,
but there are drawbacks to this too, as the type can be ambiguous in a very
large number of cases without a label.

Why are you using the JSON encoding for Avro?  The primary purpose of the
JSON serialization form as it is now is for transforming the binary to human
readable form. 
Instead of building your GenericRecord from a JSON string, try using


On 4/5/13 4:59 AM, "Jonathan Coveney" <jcoveney@gmail.com> wrote:

> Ok, I figured out the issue:
> If you make string c the following:
> String c = "{\"name\": \"Alyssa\", \"favorite_number\": {\"int\": 256},
> \"favorite_color\": {\"string\": \"blue\"}}";
> Then this works.
> This represents a divergence between the python and the Java implementation...
> the above does not work in Python, but it does work in Java. And of course,
> vice versa.
> I think I know how to fix this (and can file a bug with my reproduction and
> the fix), but I'm not sure which one is the expected case? Which
> implementation is wrong?
> Thanks
> 2013/4/5 Jonathan Coveney <jcoveney@gmail.com>
>> Correction: the issue is when reading the string according to the avro
>> schema, not on writing. it fails before I get a chance to write :)
>> 2013/4/5 Jonathan Coveney <jcoveney@gmail.com>
>>> I implemented essentially the Java avro example but using the
>>> GenericDatumWriter and GenericDatumReader and hit an issue.
>>> https://gist.github.com/jcoveney/5317904
>>> This is the error:
>>> Exception in thread "main" java.lang.RuntimeException:
>>> org.apache.avro.AvroTypeException: Expected start-union. Got
>>>     at com.spotify.hadoop.mapred.Hrm.main(Hrm.java:45)
>>> Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got
>>>     at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)
>>>     at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)
>>>     at 
>>> org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
>>>     at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>>>     at 
>>> org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
>>>     at 
>>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
>>>     at 
>>> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.jav
>>> a:177)
>>>     at 
>>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
>>>     at 
>>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
>>>     at com.spotify.hadoop.mapred.Hrm.main(Hrm.java:38)
>>> Am I doing something wrong? Is this a bug? I'm digging in now but am curious
>>> if anyone has seen this before?
>>> I get the feeling I am working with Avro in a way that most people do not :)

View raw message