avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Schmitt <dan.schm...@gmail.com>
Subject Re: Deep polymorphic type support
Date Fri, 08 Sep 2017 20:20:59 GMT
I was thinking:

record of:
    string: id
    Union: Payload

Assuming you have some render comparable function that turns a trait
Payload into an avro chunk
(which will have a schema) it's just a matter of adding the schema to
the Payload union.  You may
have to write that part yourself (I'm playing from the C++ side, not
the scala side, and while the
binary file format is flexible, you do end up writing more code to map
it to various language features..;
I don't know what you'd do to get it to de-serialize to a trait
(probably some extra tag in the payload
schema and code somewhere in the scala API to serialize/de-serialize it.)

On Fri, Sep 8, 2017 at 4:09 PM, Greg Zoller <gzoller@gmail.com> wrote:
> Thanks, Dan.  I'm not 100% clear how Union would work for Scala marker
> traits.  I could see how I'd use them for Message[Alarm], where Alarm is a
> first-class object.  I've got 2 layers of marker traits in the middle tho:
> Message[PayloadKind], where PayloadKind is a trait having no fields, Event
> is a trait having no fields that extends PayloadKind, and finally Alarm
> (concrete class) extends Event.
> I guess for pure Avro you could flatten things to just Message[Alarm], but
> this weakens the domain model.
> Can you think of a way to use Union types to represent marker traits?  My
> understanding is Unions can't immediately contain other Unions (i.e. like
> Event -> PayloadKind).
> Greg
> On Fri, Sep 8, 2017 at 1:56 PM, Dan Schmitt <dan.schmitt@gmail.com> wrote:
>> Sounds like a job for a Union for payload.
>> And if the polymorphic payload is avro serialized you don't
>> have to know the schema ahead of time, you just need a
>> way to merge the new payload schema into the union as
>> you get it (so you can push the schema responsibility to
>> the client code giving you the event.)
>> I've been futzing with something similar on the C++ avro side
>> and the provided API needs some tweaks to make it go (I'm
>> looking at moving the fileReader/Wrtier up to work with
>> streams, then you can do memory array with the array stream
>> adapter; and then you need a way to get to addField for
>> the UnionSchema (which on the C++ side only has default
>> constructor, no way to access a Union somewhere in a schema
>> and have addField add the schema from the new data
>> without knowing a lot about the internals, so you either have
>> to save it when first creating the schema, and can't do it from
>> an existing schema, or need more code.)
>>      Dan S.
>> On Fri, Sep 8, 2017 at 2:35 PM, Greg Zoller <gzoller@gmail.com> wrote:
>> > Hello,
>> >
>> > I'm trying to learn if Avro currently supports deep polymorphic types
>> > like
>> > outlined here:
>> >
>> > https://stackoverflow.com/questions/46120578/how-might-i-represent-a-scala-trait-in-avro
>> >
>> > If not, could the standard be extended?
>> >
>> > The use case is this... I want to send an AlarmMessage, serialized as a
>> > Message[PayloadKind].  The receiving side would have enough type data to
>> > reconstruct the AlarmMessage (Alarm is of type Event, which is of type
>> > PayloadKind).  In other words I'd "listen" for Message[PayloadKind] and
>> > be
>> > handed an AlarmMessage.
>> >
>> > Of course we'd need to have schema entries for any kind of Event (or
>> > PayloadKind child for that matter), but there'd also need to be a way to
>> > represent these intermediate marker traits.
>> >
>> > Thoughts?
>> > Greg
>> >
>> > (A trivial json example of what I mean can be found here:
>> > https://github.com/gzoller/ScalaJack/blob/master/doc/classesAndTraits.md
>> > )

View raw message