avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-816) Schema Comparison Utils
Date Thu, 14 Jul 2011 20:42:59 GMT

    [ https://issues.apache.org/jira/browse/AVRO-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065520#comment-13065520
] 

Scott Carey commented on AVRO-816:
----------------------------------

{quote}
Maybe 'canRead/readsBoth'? 
{quote}

Lets not introduce the notion of reading, writing, or serializing at all in Schema.java. 
'subsumes' may not be the best term, but whatever term is used should not relate to things
outside of the scope of unions.

As part of my conceptual work on AVRO-859 I realized that schema resolution/promotion does
not only apply to reads.  A resolution applies from a schema 'source' to a schema 'target'.
 The source could be an object graph, with the target another object graph or the source could
be bytes, and the target bytes.  One could have an object graph representing one schema and
project out only a subset schema on write.  

> Schema Comparison Utils
> -----------------------
>
>                 Key: AVRO-816
>                 URL: https://issues.apache.org/jira/browse/AVRO-816
>             Project: Avro
>          Issue Type: New Feature
>          Components: java
>            Reporter: Joe Crobak
>            Assignee: Joe Crobak
>            Priority: Minor
>         Attachments: AVRO-816.patch, AVRO-816.patch, AVRO-816.patch, AVRO-816.patch
>
>
> From my post on the mailing list, and Doug's response:
> {quote}
> On 05/05/2011 10:29 AM, Joe Crobak wrote:
> > We've recently come across a situation where we have two data files with
> > different schemas that we'd like to process together using
> > GenericDatumReader.  One schema is promotable to the other, but not vice
> > versa.  We'd like to programmatically determine which of the schemas to
> > use.  I did a brief look through javadoc and tests, and I couldn't find
> > any examples of checking if one schema is promotable to the other.  Has
> > anyone else come across this?
> >
> > For some context, we're considering patching AvroStorage [1] to remove
> > the assumption that all files have the same schema.  In our case, our
> > schema has evolved in that a field that was an int was promoted to a long.
> A boolean method that tells you if one schema is promotable to another
> would work in this case, but would not help in cases where, e.g.,
> different fields had changed in different versions.  For example, in
> branched development, two branches might each add a distinct symbol to
> an enum.  So I think you might be better off with a method that, given
> two schemas, returns their superset, a schema that can read data written
> by either.
> Such a method does not yet exist in Avro, but should not be difficult to
> add.  Please file an issue in Jira if this sounds of interest.
> Doug
> {quote}
> I think it would be useful to have both of the methods that Doug mentioned in some sort
of schema utils class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message