avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-816) Schema Comparison Utils
Date Tue, 05 Jul 2011 18:05:16 GMT

    [ https://issues.apache.org/jira/browse/AVRO-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060048#comment-13060048
] 

Doug Cutting commented on AVRO-816:
-----------------------------------

Patch looks great.  A few things we might consider changing before committing:
 - The relationship between 'isSuperset' and 'unify' might be a bit clearer if different names
were used.  'isSuperset/union' would be confusing since 'union' already has a meaning in Avro,
so maybe 'subsumes/unify' would be slightly better?  What do you think?
 - Unification currently ignores schema & record properties and field ordering.  This
is fine, since they're not needed for reading, but probably should be called out in the javadoc.
 Someone could add this as a feature later if they need it.
 - When default values differ maybe an exception should be thrown?
 - Should a call to unifyAliases() be added to RecordSchema? 
 - I would prefer that the methods instead started like: {code}
  if (type != RECORD)
    return super.unify(other);
  ... {code} This removes a level of indentation from the bulk of the method and makes it
clear to the reader that no other cases are dealt with.

> Schema Comparison Utils
> -----------------------
>
>                 Key: AVRO-816
>                 URL: https://issues.apache.org/jira/browse/AVRO-816
>             Project: Avro
>          Issue Type: New Feature
>          Components: java
>            Reporter: Joe Crobak
>            Assignee: Joe Crobak
>            Priority: Minor
>         Attachments: AVRO-816.patch, AVRO-816.patch
>
>
> From my post on the mailing list, and Doug's response:
> {quote}
> On 05/05/2011 10:29 AM, Joe Crobak wrote:
> > We've recently come across a situation where we have two data files with
> > different schemas that we'd like to process together using
> > GenericDatumReader.  One schema is promotable to the other, but not vice
> > versa.  We'd like to programmatically determine which of the schemas to
> > use.  I did a brief look through javadoc and tests, and I couldn't find
> > any examples of checking if one schema is promotable to the other.  Has
> > anyone else come across this?
> >
> > For some context, we're considering patching AvroStorage [1] to remove
> > the assumption that all files have the same schema.  In our case, our
> > schema has evolved in that a field that was an int was promoted to a long.
> A boolean method that tells you if one schema is promotable to another
> would work in this case, but would not help in cases where, e.g.,
> different fields had changed in different versions.  For example, in
> branched development, two branches might each add a distinct symbol to
> an enum.  So I think you might be better off with a method that, given
> two schemas, returns their superset, a schema that can read data written
> by either.
> Such a method does not yet exist in Avro, but should not be difficult to
> add.  Please file an issue in Jira if this sounds of interest.
> Doug
> {quote}
> I think it would be useful to have both of the methods that Doug mentioned in some sort
of schema utils class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message