Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CALEq1Z_3etOk0GfVLaSyBy0DhH4eG_wziWyYde3K97LjUBb4NA@mail.gmail.com>
References: 
 <CACcV0ZTzKhsKqoS_yuFuKwi6T1OgwhEV1Rp7z-aBQZnCA5x5HA@mail.gmail.com>
	<CALEq1Z_3etOk0GfVLaSyBy0DhH4eG_wziWyYde3K97LjUBb4NA@mail.gmail.com>
Date: Sat, 3 Nov 2012 09:48:52 -0700
Message-ID: 
 <CACcV0ZS14=G5mTO5sKUdAj-C2AwZxN7BRzFNrMP4TYu69JM0fw@mail.gmail.com>
Subject: Re: Schema validation of a field's default values
From: Mark Hayes <mark@greybird.com>
To: user@avro.apache.org
Content-Type: multipart/alternative; boundary=047d7b5daf50f8640b04cd9a06cc

--047d7b5daf50f8640b04cd9a06cc
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Oct 29, 2012 at 12:32 PM, Doug Cutting <cutting@apache.org> wrote:

> No, I don't know of a default value validator that's been implemented
> yet.  It would be great to have one.
>
> I think this would recursively walk a schema.  Whenever a non-null
> default value is found it could call ResolvingGrammarDecoder#encode().
>  That's what interprets Json default values.  (Perhaps this logic
> should be moved, though.)


Thanks for the reply Doug.

I did find ResolvingGrammarDecoder.encode (I saw that it is called by the
builders) and was using it as you described, but I ran into limitations:

+  When the field type is an array, map or record, values of the wrong JSON
type (not array or object) are translated to an empty array, map or record.
 For example, specifying a default of 0, null or "" results in an empty
array, map or record.

+ For all numeric Avro types (int, long, float and double) the default
value may be of any JSON numeric type, and the JSON values will be coerced
to the Avro type in spite of the fact that part of the value may be
lost/truncated.  For example, a long default value that exceeds 32-bits
will be truncated if the field is type int.

+ The byte array length is not validated for a fixed type.

+ For nested fields and certain types (e.g., enums) a cryptic error
is often output that does not contain the name of the offending field.

These deficiencies can mask errors made by the user when defining a default
value.  This is important to our application.

To compensate for these deficiencies we implemented our own checking that
is more strict than Avro's.  To do this, we serialize the default value
using our own JSON serializer in a special mode where default values are
applied.  Any errors during serialization indicate that the default value
is invalid.

Something similar might be done in Avro itself, for example, if the JSON
encoder were made to operate in a special mode where default values are
applied.

--mark

--047d7b5daf50f8640b04cd9a06cc
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Oct 29, 2012 at 12:32 PM, Doug Cutting <span dir=3D"ltr">&lt;<a hre=
f=3D"mailto:cutting@apache.org" target=3D"_blank">cutting@apache.org</a>&gt=
;</span> wrote:<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_qu=
ote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex=
">

<div>No, I don&#39;t know of a default value validator that&#39;s been impl=
emented</div>
yet. =A0It would be great to have one.<br>
<br>
I think this would recursively walk a schema. =A0Whenever a non-null<br>
default value is found it could call ResolvingGrammarDecoder#encode().<br>
=A0That&#39;s what interprets Json default values. =A0(Perhaps this logic<b=
r>
should be moved, though.)</blockquote><div><br></div><div>Thanks for the re=
ply Doug.</div><div><br></div><div>I did find=A0ResolvingGrammarDecoder.enc=
ode (I saw that it is called by the builders) and was using it as you descr=
ibed, but I ran into limitations:</div>

<div><br></div><div><div>+ =A0When the field type is an array, map or recor=
d, values of the wrong=A0JSON type (not array or object) are translated to =
an empty array, map=A0or record. =A0For example, specifying a default of 0,=
 null or &quot;&quot; results in an empty array, map or record.</div>
<div><br></div><div>+ For all numeric Avro types (int, long, float and doub=
le) the default=A0 value may be of any JSON numeric type, and the JSON valu=
es will be=A0coerced to the Avro type in spite of the fact that part of the=
 value=A0may be lost/truncated. =A0For example, a long default value that e=
xceeds 32-bits will be truncated if the field is type int.</div>
<div><br></div><div>+ The byte array length is not validated for a fixed ty=
pe.</div><div><br></div><div>+ For nested fields and certain types (e.g., e=
nums) a cryptic error is=A0often output that does not contain the name of t=
he offending field.</div>
<div><br></div><div>These deficiencies can mask errors made by the user whe=
n defining a=A0default value. =A0This is important to our application.</div=
><div><br></div><div>To compensate for these deficiencies we implemented ou=
r=A0own checking that is more strict than Avro&#39;s. =A0To do this, we ser=
ialize=A0the default value using our own JSON serializer in a special mode =
where=A0default values are applied. =A0Any errors during serialization indi=
cate=A0that the default value is invalid.</div>
</div><div><br></div><div>Something similar might be done in Avro itself, f=
or example, if the JSON encoder were made to operate in a special mode wher=
e default values are applied.</div><div><br></div><div>--mark</div></div>


--047d7b5daf50f8640b04cd9a06cc--