Mailing-List: contact dev-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@avro.apache.org
MIME-Version: 1.0
Date: Mon, 1 Feb 2016 12:31:25 -0800
Message-ID: 
 <CAKKkEq_CWmgMT8_u1pfFuX1JR5g7me7V7q4B1tmY8pu8MjK6iQ@mail.gmail.com>
Subject: Avro schema doesn't honor backward compatibilty
From: Raghvendra Singh <rsingh@appdynamics.com>
To: dev@avro.apache.org, user@avro.apache.org
Content-Type: multipart/alternative; boundary=001a11412f5cdad708052abb44ae

--001a11412f5cdad708052abb44ae
Content-Type: text/plain; charset=UTF-8

down votefavorite
<http://stackoverflow.com/questions/34733604/avro-schema-doesnt-honor-backward-compatibilty#>

I have this avro schema

{
 "namespace": "xx.xxxx.xxxxx.xxxxx",
 "type": "record",
 "name": "MyPayLoad",
 "fields": [
     {"name": "filed1",  "type": "string"},
     {"name": "filed2",     "type": "long"},
     {"name": "filed3",  "type": "boolean"},
     {
          "name" : "metrics",
          "type":
          {
             "type" : "array",
             "items":
             {
                 "name": "MyRecord",
                 "type": "record",
                 "fields" :
                     [
                       {"name": "min", "type": "long"},
                       {"name": "max", "type": "long"},
                       {"name": "sum", "type": "long"},
                       {"name": "count", "type": "long"}
                     ]
             }
          }
     }
  ]}

Here is the code which we use to parse the data

public static final MyPayLoad parseBinaryPayload(byte[] payload) {
        DatumReader<MyPayLoad> payloadReader = new
SpecificDatumReader<>(MyPayLoad.class);
        Decoder decoder = DecoderFactory.get().binaryDecoder(payload, null);
        MyPayLoad myPayLoad = null;
        try {
            myPayLoad = payloadReader.read(null, decoder);
        } catch (IOException e) {
            logger.log(Level.SEVERE, e.getMessage(), e);
        }

        return myPayLoad;
    }

Now i want to add one more field int the schema so the schema looks like
below

 {
 "namespace": "xx.xxxx.xxxxx.xxxxx",
 "type": "record",
 "name": "MyPayLoad",
 "fields": [
     {"name": "filed1",  "type": "string"},
     {"name": "filed2",     "type": "long"},
     {"name": "filed3",  "type": "boolean"},
     {
          "name" : "metrics",
          "type":
          {
             "type" : "array",
             "items":
             {
                 "name": "MyRecord",
                 "type": "record",
                 "fields" :
                     [
                       {"name": "min", "type": "long"},
                       {"name": "max", "type": "long"},
                       {"name": "sum", "type": "long"},
                       {"name": "count", "type": "long"}
                     ]
             }
          }
     }
     {"name": "agentType",  "type": ["null", "string"], "default": "APP_AGENT"}
  ]}

Note the filed added and also the default is defined. The problem is that
if we receive the data which was written using the older schema i get this
error

java.io.EOFException: null
    at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:423)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
~[avro-1.7.4.jar:1.7.4]
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
~[avro-1.7.4.jar:1.7.4]
    at com.appdynamics.blitz.shared.util.XXXXXXXXXXXXX.parseBinaryPayload(BlitzAvroSharedUtil.java:38)
~[blitz-shared.jar:na]

What i understood from this
<https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html>
document
that this should have been backward compatible but somehow that doesn't
seem to be the case. Any idea what i am doing wrong?

--001a11412f5cdad708052abb44ae--