avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@gmail.com>
Subject Re: Avro schema resolution
Date Fri, 01 Sep 2017 20:19:53 GMT
This seems like a bug in the Java implementation, not a mistake in the
spec.  Int->float promotion is supported elsewhere and, while lossy, has
always been in the spec.  There are other lossy promotions (e.g.,
long->float) and dropping missing record fields is lossy, so lossiness
alone doesn't seem a sufficient criterion to disallow this promotion.

Doug

On Fri, Sep 1, 2017 at 6:32 AM, Sean Busbey <busbey@cloudera.com> wrote:

> What's the rationale for not allowing it at all? Given that it's been in
> the spec so long I'm hesitant to remove it.
>
> I can't think of a time that we did a breaking change to the spec. Even if
> we check all of our implementations, there's no way to know if someone else
> has already created an implementation that allows the promotion since it's
> in the spec.
>
> On Wed, Aug 30, 2017 at 2:45 AM, Nandor Kollar <nkollar@cloudera.com>
> wrote:
>
> > Thanks Doug! In this case, I think in this case ResolvingGrammarGenerator
> > doesn't implement the schema resolution logic according to the spec,
> > created a Jira for it:
> >
> > https://issues.apache.org/jira/browse/AVRO-2072
> >
> > What do you think about Anders comment about to int -> float promotion
> > on AVRO-1931?
> > Should Avro allow it at all?
> >
> > Nandor
> >
> > On Wed, Aug 30, 2017 at 1:53 AM, Doug Cutting <cutting@gmail.com> wrote:
> >
> > > I'm not sure why both SchemaCompatibility and SchemaValidator exist.  I
> > > think folks perhaps just didn't realize the former existed when the
> > latter
> > > was added.
> > >
> > > https://issues.apache.org/jira/browse/AVRO-1409
> > > https://issues.apache.org/jira/browse/AVRO-1315
> > >
> > > The latter looks like a probably more efficient implementation, since
> it
> > > doesn't compile a parser for each schema validated, but the former
> shares
> > > logic with the reading & writing of data, simplifying maintenance.
> > >
> > > Doug
> > >
> > > On Wed, Aug 23, 2017 at 6:01 AM, Nandor Kollar <nkollar@cloudera.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > While I was trying to understand how schema evolution works in Avro,
> I
> > > > noticed that the schema resolution logic (
> > > > https://avro.apache.org/docs/1.8.1/spec.html#Schema+Resolution) is
> > > > implemented in two different places in Avro: in
> > ResolvingGrammarGenerator
> > > > and in SchemaCompatibility.
> > > >
> > > > Could someone please explain the differences between
> > > > ResolvingGrammarGenerator, SchemaCompatibility, and SchemaValidator?
> It
> > > > seems that SchemaValidator is used to check the compatibility between
> > the
> > > > reader and the writer schema, which sounds exactly what
> > > SchemaCompatibility
> > > > does, why does it use ResolvingGrammarGenerator instead of
> > > > SchemaCompatibility? Why is the same schema resolution logic
> > implemented
> > > > twice (if I understood correctly)?
> > > >
> > > > I also have a question related to unions and schema resolution. In
> the
> > > > patch for AVRO-1933 Anders commented out a case for unions: union
> with
> > an
> > > > int branch is not compatible with a long union reader. After a brief
> > > > investigation I noticed, that ResolvingGrammarGenerator#bestBranch
> > > doesn't
> > > > allow int -> float promotion in union branches. Is this a bug? Should
> > we
> > > > allow this promotion at all, since like Anders mentioned in AVRO-1931
> > > this
> > > > is a lossy conversion? What do you think?
> > > >
> > > > Thanks,
> > > > Nandor
> > > >
> > >
> >
>
>
>
> --
> busbey
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message