avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Burkert <danburk...@gmail.com>
Subject Re: Effort towards Avro 2.0?
Date Wed, 04 Dec 2013 01:41:00 GMT
I am mistaken in the previous email-- specific record does implement the
generic record API.  However, generated specific record builders are not
generic, so building a specific record with fields not known at compile
time requires reflection.

- Dan


On Tue, Dec 3, 2013 at 5:28 PM, Dan Burkert <danburkert@gmail.com> wrote:

> It would be nice if specific records implemented the generic record API.
>  This is useful when you don't know field names of a specific record at
> compile time.  Reflection is the only way around this currently.  Also, it
> would be useful if there was built in support for determining if a given
> schema is reader-compatible with a writer-schema.  Neither of these should
> require changing the data serialization format.
>
> - Dan
>
> On Tue, Dec 3, 2013 at 7:20 AM, Doug Cutting <cutting@apache.org> wrote:
>
>> On Mon, Dec 2, 2013 at 1:56 PM, Philip Zeyliger <philip@cloudera.com>
>> wrote:
>> > It sounds like you're proposing to break language API compatibility.
>>  Are
>> > you also proposing to break wire compatibility for Avro HTTP RPC, Avro
>> Netty
>> > RPC, and/or Avro datafiles?
>>
>> We should be able to provide back-compatibility.  When current APIs
>> cannot be back-compatibly extended, new APIs can be added.  Old APIs
>> may be deprecated but should be retained for a while.  Data files
>> written by 1.x should be readable by 2.x.
>>
>> Forward compatibility may not be possible when new schema features are
>> used.  Data files written in 2.x may not be readable by 1.x.  Perhaps
>> we could add a mode that forces 2.x to write a 1.x format file.
>>
>> RPC interoperability requires that 2.x be able to both read and write
>> 1.x format.  So long as a 1.x protocol is used, then 1.x and 2.x
>> clients and servers might be able to interoperate using 1.x wire
>> formats.  But when 2.x schema features are used this may not be
>> possible.
>>
>> Perhaps we should proceed by making back-compatibility (ability to
>> read 1.x) a requirement, then adding interoperabilty features (ability
>> to write 1.x) as needed?
>>
>> Should we require that all new schema features (named unions,
>> extensions, date primitive, etc,) have a lossless translation to a 1.x
>> schema?
>>
>> Doug
>>
>
>

Mime
View raw message