avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <sc...@richrelevance.com>
Subject Re: "include" ability in schema file
Date Wed, 01 Jun 2011 17:08:20 GMT
Yes, I have had to do some automated copy-paste myself.

You could also put Blah and Bar into idl form.

The Avro project would welcome patches to the compilers to fix these sorts of issues.  Most
of the Java APIs handle these cases, but the code that reads files and converts them to Schema
objects is fairly simple.

On 6/1/11 9:56 AM, "Yang" <teddyyyy123@gmail.com<mailto:teddyyyy123@gmail.com>>

actually the link that Doug gave shows that idl can indeed import avpr/avsc

back to the record reference problem:  yes I can reference Foo from Bar, but since I put Bar
and let's say Blah , outside of my idl,
they are independent .avsc now, if Bar and Blah both references Foo, where should I put Foo?
I can't reference another .avsc from Bar.avsc or Blah.avsc,
so I have to copy-paste Foo in Blah.avsc and Bar.avsc

the copy-paste can be trivially automated with some perl scripts, it's just a bit hacky


On Wed, Jun 1, 2011 at 9:48 AM, Scott Carey <scott@richrelevance.com<mailto:scott@richrelevance.com>>
In an *.avsc you can have multiple records in a JSON array:
 {"type":"record", "name":"Foo", … } ,
 {"type":"record", "name":"Bar", … }
And in the above case, Bar can reference Foo.

idl can import other idl's but not an avsc or avpr to my knowledge.

On 6/1/11 8:57 AM, "Yang" <teddyyyy123@gmail.com<mailto:teddyyyy123@gmail.com>>

Thanks Doug.

this .idl form provides a great tool to better organize my schemas.

but a slightly more complicated use case of mine still has problems:

let's say my hello.idl reference record type "Orange", so I define "Orange" record type in
but Orange contains a Fruit record, so I need to define a Fruit.avsc and reference it from
but .avsc does not have the import ability


On Wed, Jun 1, 2011 at 1:06 AM, Doug Cutting <cutting@apache.org<mailto:cutting@apache.org>>
Avro IDL supports imports:


One can also use a protocol with no messages to define multiple,
dependent types.

To get a type from a protocol file in Java one can use something like:

 Schema s = Protocol.parse("MyProtocol.avpr").getType("Foo");


On 06/01/2011 01:33 AM, Yang wrote:
> I have a protocol schema  hello.avpr
> { ....
> types: {
>    { name : input_msg ,  type: record : fields [
>         { name: date , type int },
>        { name :msg, type: string}
>     ]
>   }
> }
> }
> now if I need to serialize the record "input_msg", I'm going to need its
> schema,
> so I'd need to pass a schema obj to DataFileWriter. but how can I parse
> the schema out from schema file?
> the schema for "input_msg" is not a standalone file.
> I would like a way to write the type of "input_msg" in a standalone
> file, and then reference/include that from my hello.avpr.
> is this possible?
> also btw, for an avro-generate object class, would it be better to make
> the getSchema() method static?
> that way I can get its schema without creating an actual obj
> maybe this is not a big issue, we can simply create some hack scripts to
> paste together individual files, i.e. creating our own
> "include" macro, but that's kind of kludgy
> thanks
> Yang

View raw message