avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Camp, Jonathan" <jonathan.c...@bluecoat.com>
Subject avdl schema compatibility
Date Sun, 26 Oct 2014 19:20:59 GMT
Hi All, I am trying to use avro's protocol definition to provide the basis for an RPC system.
In this case, I'll be handling the handshaking and transport layer rather the avro's IPC package.
Are the schemas generated from the protocol files compatible with the standard DataFileWriter/Reader
schemas (I am using python3)? If I try to load a file generated from an avdl message (using
DataFileWriter(bytes, DatumWriter(), protocol.message_map['hash'].request)) then when I try
to load it, I receive error messages stating that the writer_schema is essentially invalid
(see stacktrace below).

Here is a code snippet that seems to show an invalid schema being generated. I think its because
the value of the 'type' field (for objfsptr) is a python/json dictionary rather that the 'record'
type i think its supposed to be.

import avro.protocol
p = avro.protocol.Parse(open("service.avpr").read())

[{'name': 'objfsptr',
  'type': {'fields': [{'name': 'uri', 'type': 'string'}],
           'name': 'ObjFSPtr',
           'namespace': 'FooServiceType',
           'type': 'record'}},
 {'default': 10, 'name': 'x', 'type': 'int'}]

Stacktrace from a deserialization attempt:

Traceback (most recent call last):
  File "./schema_test.py", line 74, in <module>
    requests = deserializer.deserialize('hash', srequest)
  File "/data/bluecoat/services.py", line 123, in deserialize
    reader = DataFileReader(io.BytesIO(raw_bytes), DatumReader())
  File "/data/avro/datafile.py", line 363, in __init__
  File "/data/avro/schema.py", line 1283, in Parse
    return SchemaFromJSONData(json_data, names)
  File "/data/avro/schema.py", line 1254, in SchemaFromJSONData
    return parser(json_data, names=names)
  File "/data/avro/schema.py", line 1142, in _SchemaFromJSONArray
    return UnionSchema(map(MakeSchema, json_array))
  File "/data/avro/schema.py", line 866, in __init__
    self._schemas = tuple(schemas)
  File "/data/avro/schema.py", line 1141, in MakeSchema
    return SchemaFromJSONData(json_data=desc, names=names)
  File "/data/avro/schema.py", line 1254, in SchemaFromJSONData
    return parser(json_data, names=names)
  File "/data/avro/schema.py", line 1154, in _SchemaFromJSONObject
    if type in PRIMITIVE_TYPES:
TypeError: unhashable type: 'dict'


protocol MyProtocol {

    record ObjFSPtr {
        string uri;

    string hash(FooServiceType.ObjFSPtr objfsptr, int x=10);

generated service.avpr:

  "protocol" : "MyProtocol",
  "namespace" : "FooService",
  "types" : [ {
    "type" : "record",
    "name" : "ObjFSPtr",
    "namespace" : "FooServiceType",
    "fields" : [ {
      "name" : "uri",
      "type" : "string"
    } ]
  } ],
  "messages" : {
    "hash" : {
      "request" : [ {
        "name" : "objfsptr",
        "type" : "FooServiceType.ObjFSPtr"
      }, {
        "name" : "x",
        "type" : "int",
        "default" : 10
      } ],
      "response" : "string"


Jonathan Camp

View raw message