avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Maes <jm...@linkedin.com>
Subject Re: ArrayIndexOutOfBoundsException while writing an old sub record to an evolved superrecord
Date Wed, 25 Mar 2015 20:48:55 GMT
Still troubleshooting this and I noticed I get the same exception even if I skip the Json encoding
and just execute:

GenericData.get().validate(sinkFieldSchema, override)

Where “sinkFieldSchema" is the schema of the field within the evolved persistence schema
which corresponds to the message schema and “override” is the value of the message.

I wouldn’t think validate() should throw. Is it a bug?  More importantly, is there an alternative
way to insert sub records into a super record that accounts for schema evolution?


From: Jacob Maes <jmaes@linkedin.com<mailto:jmaes@linkedin.com>>
Date: Tuesday, March 24, 2015 at 6:32 PM
To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Subject: ArrayIndexOutOfBoundsException while writing an old sub record to an evolved superrecord


I’m currently working on a system that receives Avro messages via Kafka which contain message
records and then writes them to a persistent store using a schema that is a super record (contains
all the schemas of the message records).

Recently, I have had to evolve the persistence schema to include a couple extra (optional)
fields. After making this schema change, I get an ArrayIndexOutOfBoundsException when the
system receives an old version of the message and tries to inject it into the new version
of the schema.

I’m attaching a test case that demonstrates the issue. The testAvroNormalUpdate() case passes
while the testAvroEvolvedUpdate() case produces the stack trace below.

The new schema should be backward compatible, so I’m not sure where I went wrong.  I’m
using Avro 1.4, but I’ve confirmed the same issue on Avro 1.7.7.

Any help would be much appreciated.


java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.avro.generic.GenericData$Record.get(GenericData.java:64)
at org.apache.avro.generic.GenericDatumWriter.getField(GenericDatumWriter.java:113)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:100)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:133)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:56)
at com.pub.avro.TestEvolvedSchema.genericRecordToJson(TestEvolvedSchema.java:234)
at com.pub.avro.TestEvolvedSchema.genericRecordToJson(TestEvolvedSchema.java:240)
at com.pub.avro.TestEvolvedSchema.testAvroEvolvedUpdate(TestEvolvedSchema.java:190)

View raw message