Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
MIME-Version: 1.0
From: Vinicius Carvalho <viniciusccarvalho@gmail.com>
Date: Wed, 4 May 2016 11:02:14 -0400
Message-ID: <CAFPzsfSvUKZyZbK=Mp9CSCm=z+9br_KdozN=Dc7Wa9yXdC0Yog@mail.gmail.com>
Subject: Schema resolution question
To: user@avro.apache.org
Content-Type: multipart/alternative; boundary=94eb2c053b64f2c8550532058381
archived-at: Wed, 04 May 2016 15:02:38 -0000

--94eb2c053b64f2c8550532058381
Content-Type: text/plain; charset=UTF-8

Hi there, I'm using Avro 1.8.0, and I'm not sure why one of my
compatibility tests is failing.

I have two schemas:

V1:

{
  "namespace" : "io.igx.android",
  "type" : "record",
  "name" : "Sensor",
  "fields" : [
    {"name":"id","type":"string"},
    {"name":"temperature", "type":"float"},
    {"name":"acceleration", "type":"float","default":0.0},
    {"name":"velocity","type":"float","default":0.0},
    {"name":"accelerometer","type":[
      "null",{
        "type":"array",
        "items":"float"
      }
    ]},
    {"name":"magneticField","type":[
      "null",{
        "type":"array",
        "items":"float"
      }
    ]},
    {"name":"orientation","type":[
      "null",{
        "type":"array",
        "items":"float"
      }
    ],"default":"null"}

  ]

}

V2:
{
  "namespace" : "io.igx.android",
  "type" : "record",
  "name" : "Sensor",
  "fields" : [
    {"name":"id","type":"string"},
    {"name":"temperature", "type":"float", "default":0.0},
    {"name":"acceleration", "type":"float","default":0.0},
    {"name":"velocity","type":"float","default":0.0},
    {"name":"accelerometer","type":[
      "null",{
        "type":"array",
        "items":"float"
      }
    ]},
    {"name":"magneticField","type":[
      "null",{
        "type":"array",
        "items":"float"
      }
    ]}

  ]

}

As you can see the only difference between those two is that one of the
arrays is removed from V2, and I've added a default value to a property on
V2.

If I run:
SchemaCompatibility.checkReaderWriterCompatibility(load("schemas/sensor_v2.avsc"),load("schemas/sensor_v1.avsc")).getType()

It prints out that data written in v2 should be able to be read by v1,
which seems logical

But when I try to deserialize data from v2 using v1 reader I get an error:

Sensor v1Fromv2 = deserialize(v2Bytes,v1.getSchema(),v2.getSchema());

private <T> T deserialize(byte[] bytes, Schema reader, Schema writer)
throws Exception{
DatumReader datumReader = new SpecificDatumReader<>(writer,reader);
Decoder decoder = DecoderFactory.get().binaryDecoder(bytes,null);
return (T) datumReader.read(null,decoder);
}
org.apache.avro.AvroTypeException: Non-null default value for null type:
"null"

Not sure why is the deserialization not working if the compatibility check
passes

Any ideas?

Regards

--94eb2c053b64f2c8550532058381
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi there, I&#39;m using Avro 1.8.0, and I&#39;m not sure w=
hy one of my compatibility tests is failing.<div><br></div><div>I have two =
schemas:</div><div><br></div><div>V1:</div><div><br></div><div><div>{</div>=
<div>=C2=A0 &quot;namespace&quot; : &quot;io.igx.android&quot;,</div><div>=
=C2=A0 &quot;type&quot; : &quot;record&quot;,</div><div>=C2=A0 &quot;name&q=
uot; : &quot;Sensor&quot;,</div><div>=C2=A0 &quot;fields&quot; : [</div><di=
v>=C2=A0 =C2=A0 {&quot;name&quot;:&quot;id&quot;,&quot;type&quot;:&quot;str=
ing&quot;},</div><div>=C2=A0 =C2=A0 {&quot;name&quot;:&quot;temperature&quo=
t;, &quot;type&quot;:&quot;float&quot;},</div><div>=C2=A0 =C2=A0 {&quot;nam=
e&quot;:&quot;acceleration&quot;, &quot;type&quot;:&quot;float&quot;,&quot;=
default&quot;:0.0},</div><div>=C2=A0 =C2=A0 {&quot;name&quot;:&quot;velocit=
y&quot;,&quot;type&quot;:&quot;float&quot;,&quot;default&quot;:0.0},</div><=
div>=C2=A0 =C2=A0 {&quot;name&quot;:&quot;accelerometer&quot;,&quot;type&qu=
ot;:[</div><div>=C2=A0 =C2=A0 =C2=A0 &quot;null&quot;,{</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 &quot;type&quot;:&quot;array&quot;,</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 &quot;items&quot;:&quot;float&quot;</div><div>=C2=A0 =
=C2=A0 =C2=A0 }</div><div>=C2=A0 =C2=A0 ]},</div><div>=C2=A0 =C2=A0 {&quot;=
name&quot;:&quot;magneticField&quot;,&quot;type&quot;:[</div><div>=C2=A0 =
=C2=A0 =C2=A0 &quot;null&quot;,{</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 &quo=
t;type&quot;:&quot;array&quot;,</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot=
;items&quot;:&quot;float&quot;</div><div>=C2=A0 =C2=A0 =C2=A0 }</div><div>=
=C2=A0 =C2=A0 ]},</div><div>=C2=A0 =C2=A0 {&quot;name&quot;:&quot;orientati=
on&quot;,&quot;type&quot;:[</div><div>=C2=A0 =C2=A0 =C2=A0 &quot;null&quot;=
,{</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;type&quot;:&quot;array&quot;=
,</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;items&quot;:&quot;float&quot;=
</div><div>=C2=A0 =C2=A0 =C2=A0 }</div><div>=C2=A0 =C2=A0 ],&quot;default&q=
uot;:&quot;null&quot;}</div><div><br></div><div>=C2=A0 ]</div><div><br></di=
v><div>}</div></div><div><br></div><div>V2:</div><div><div>{</div><div>=C2=
=A0 &quot;namespace&quot; : &quot;io.igx.android&quot;,</div><div>=C2=A0 &q=
uot;type&quot; : &quot;record&quot;,</div><div>=C2=A0 &quot;name&quot; : &q=
uot;Sensor&quot;,</div><div>=C2=A0 &quot;fields&quot; : [</div><div>=C2=A0 =
=C2=A0 {&quot;name&quot;:&quot;id&quot;,&quot;type&quot;:&quot;string&quot;=
},</div><div>=C2=A0 =C2=A0 {&quot;name&quot;:&quot;temperature&quot;, &quot=
;type&quot;:&quot;float&quot;, &quot;default&quot;:0.0},</div><div>=C2=A0 =
=C2=A0 {&quot;name&quot;:&quot;acceleration&quot;, &quot;type&quot;:&quot;f=
loat&quot;,&quot;default&quot;:0.0},</div><div>=C2=A0 =C2=A0 {&quot;name&qu=
ot;:&quot;velocity&quot;,&quot;type&quot;:&quot;float&quot;,&quot;default&q=
uot;:0.0},</div><div>=C2=A0 =C2=A0 {&quot;name&quot;:&quot;accelerometer&qu=
ot;,&quot;type&quot;:[</div><div>=C2=A0 =C2=A0 =C2=A0 &quot;null&quot;,{</d=
iv><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;type&quot;:&quot;array&quot;,</di=
v><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 &quot;items&quot;:&quot;float&quot;</div=
><div>=C2=A0 =C2=A0 =C2=A0 }</div><div>=C2=A0 =C2=A0 ]},</div><div>=C2=A0 =
=C2=A0 {&quot;name&quot;:&quot;magneticField&quot;,&quot;type&quot;:[</div>=
<div>=C2=A0 =C2=A0 =C2=A0 &quot;null&quot;,{</div><div>=C2=A0 =C2=A0 =C2=A0=
 =C2=A0 &quot;type&quot;:&quot;array&quot;,</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 &quot;items&quot;:&quot;float&quot;</div><div>=C2=A0 =C2=A0 =C2=A0 }=
</div><div>=C2=A0 =C2=A0 ]}</div><div><br></div><div>=C2=A0 ]</div><div><br=
></div><div>}</div></div><div><br></div><div>As you can see the only differ=
ence between those two is that one of the arrays is removed from V2, and I&=
#39;ve added a default value to a property on V2.</div><div><br></div><div>=
If I run:=C2=A0</div><div>SchemaCompatibility.checkReaderWriterCompatibilit=
y(load(&quot;schemas/sensor_v2.avsc&quot;),load(&quot;schemas/sensor_v1.avs=
c&quot;)).getType()<br></div><div><br></div><div>It prints out that data wr=
itten in v2 should be able to be read by v1, which seems logical</div><div>=
<br></div><div>But when I try to deserialize data from v2 using v1 reader I=
 get an error:</div><div><br></div><div>Sensor v1Fromv2 =3D deserialize(v2B=
ytes,v1.getSchema(),v2.getSchema());<br></div><div><br></div><div><div>priv=
ate &lt;T&gt; T deserialize(byte[] bytes, Schema reader, Schema writer) thr=
ows Exception{</div><div><span class=3D"" style=3D"white-space:pre">		</spa=
n>DatumReader datumReader =3D new SpecificDatumReader&lt;&gt;(writer,reader=
);</div><div><span class=3D"" style=3D"white-space:pre">		</span>Decoder de=
coder =3D DecoderFactory.get().binaryDecoder(bytes,null);</div><div><span c=
lass=3D"" style=3D"white-space:pre">		</span>return (T) datumReader.read(nu=
ll,decoder);</div><div><span class=3D"" style=3D"white-space:pre">	</span>}=
</div></div><div>org.apache.avro.AvroTypeException: Non-null default value =
for null type: &quot;null&quot;<br></div><div><br></div><div>Not sure why i=
s the deserialization not working if the compatibility check passes</div><d=
iv><br></div><div>Any ideas?</div><div><br></div><div>Regards</div></div>

--94eb2c053b64f2c8550532058381--