avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Maes <jm...@linkedin.com>
Subject Re: ArrayIndexOutOfBoundsException while writing an old sub record to an evolved superrecord
Date Wed, 25 Mar 2015 22:01:14 GMT
In case it helps, here’s an updated version of the test with simplified schemas (removed
Arrays, extra fields, and some union types).  This reduces some of the recursion in the stack
trace, so hopefully it will be easier to troubleshoot.

java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.avro.generic.GenericData$Record.get(GenericData.java:64)
at org.apache.avro.generic.GenericDatumWriter.getField(GenericDatumWriter.java:113)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:100)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:56)
at com.pub.avro.TestEvolvedSchema2.genericRecordToJson(TestEvolvedSchema2.java:227)
at com.pub.avro.TestEvolvedSchema2.genericRecordToJson(TestEvolvedSchema2.java:233)
at com.pub.avro.TestEvolvedSchema2.testAvroEvolvedUpdate(TestEvolvedSchema2.java:144)



From: Jacob Maes <jmaes@linkedin.com<mailto:jmaes@linkedin.com>>
Reply-To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Date: Wednesday, March 25, 2015 at 1:48 PM
To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Subject: Re: ArrayIndexOutOfBoundsException while writing an old sub record to an evolved
superrecord

Still troubleshooting this and I noticed I get the same exception even if I skip the Json
encoding and just execute:

GenericData.get().validate(sinkFieldSchema, override)

Where “sinkFieldSchema" is the schema of the field within the evolved persistence schema
which corresponds to the message schema and “override” is the value of the message.

I wouldn’t think validate() should throw. Is it a bug?  More importantly, is there an alternative
way to insert sub records into a super record that accounts for schema evolution?

Thanks,
Jake


From: Jacob Maes <jmaes@linkedin.com<mailto:jmaes@linkedin.com>>
Date: Tuesday, March 24, 2015 at 6:32 PM
To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Subject: ArrayIndexOutOfBoundsException while writing an old sub record to an evolved superrecord

Hello,

I’m currently working on a system that receives Avro messages via Kafka which contain message
records and then writes them to a persistent store using a schema that is a super record (contains
all the schemas of the message records).

Recently, I have had to evolve the persistence schema to include a couple extra (optional)
fields. After making this schema change, I get an ArrayIndexOutOfBoundsException when the
system receives an old version of the message and tries to inject it into the new version
of the schema.

I’m attaching a test case that demonstrates the issue. The testAvroNormalUpdate() case passes
while the testAvroEvolvedUpdate() case produces the stack trace below.

The new schema should be backward compatible, so I’m not sure where I went wrong.  I’m
using Avro 1.4, but I’ve confirmed the same issue on Avro 1.7.7.

Any help would be much appreciated.

Thanks,
Jake

java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.avro.generic.GenericData$Record.get(GenericData.java:64)
at org.apache.avro.generic.GenericDatumWriter.getField(GenericDatumWriter.java:113)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:100)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:133)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:56)
at com.pub.avro.TestEvolvedSchema.genericRecordToJson(TestEvolvedSchema.java:234)
at com.pub.avro.TestEvolvedSchema.genericRecordToJson(TestEvolvedSchema.java:240)
at com.pub.avro.TestEvolvedSchema.testAvroEvolvedUpdate(TestEvolvedSchema.java:190)

Mime
View raw message