avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: avdl schema compatibility
Date Mon, 27 Oct 2014 16:31:31 GMT
The type of the objfsptr parameter looks correct to me.  Rather than a
reference to the name of the record it has the record's definition.
What looks incorrect about this to you?

That said, I have no idea what's causing that error.

Doug

On Sun, Oct 26, 2014 at 12:20 PM, Camp, Jonathan
<jonathan.camp@bluecoat.com> wrote:
> Hi All, I am trying to use avro's protocol definition to provide the basis for an RPC
system. In this case, I'll be handling the handshaking and transport layer rather the avro's
IPC package. Are the schemas generated from the protocol files compatible with the standard
DataFileWriter/Reader schemas (I am using python3)? If I try to load a file generated from
an avdl message (using DataFileWriter(bytes, DatumWriter(), protocol.message_map['hash'].request))
then when I try to load it, I receive error messages stating that the writer_schema is essentially
invalid (see stacktrace below).
>
> Here is a code snippet that seems to show an invalid schema being generated. I think
its because the value of the 'type' field (for objfsptr) is a python/json dictionary rather
that the 'record' type i think its supposed to be.
>
> import avro.protocol
> p = avro.protocol.Parse(open("service.avpr").read())
> print(p.message_map['hash'].request.to_json())
>
> [{'name': 'objfsptr',
>   'type': {'fields': [{'name': 'uri', 'type': 'string'}],
>            'name': 'ObjFSPtr',
>            'namespace': 'FooServiceType',
>            'type': 'record'}},
>  {'default': 10, 'name': 'x', 'type': 'int'}]
>
>
> Stacktrace from a deserialization attempt:
>
> Traceback (most recent call last):
>   File "./schema_test.py", line 74, in <module>
>     requests = deserializer.deserialize('hash', srequest)
>   File "/data/bluecoat/services.py", line 123, in deserialize
>     reader = DataFileReader(io.BytesIO(raw_bytes), DatumReader())
>   File "/data/avro/datafile.py", line 363, in __init__
>     schema.Parse(self.GetMeta(SCHEMA_KEY).decode('utf-8')))
>   File "/data/avro/schema.py", line 1283, in Parse
>     return SchemaFromJSONData(json_data, names)
>   File "/data/avro/schema.py", line 1254, in SchemaFromJSONData
>     return parser(json_data, names=names)
>   File "/data/avro/schema.py", line 1142, in _SchemaFromJSONArray
>     return UnionSchema(map(MakeSchema, json_array))
>   File "/data/avro/schema.py", line 866, in __init__
>     self._schemas = tuple(schemas)
>   File "/data/avro/schema.py", line 1141, in MakeSchema
>     return SchemaFromJSONData(json_data=desc, names=names)
>   File "/data/avro/schema.py", line 1254, in SchemaFromJSONData
>     return parser(json_data, names=names)
>   File "/data/avro/schema.py", line 1154, in _SchemaFromJSONObject
>     if type in PRIMITIVE_TYPES:
> TypeError: unhashable type: 'dict'
>
>
>
>
> service.avdl
>
> @namespace("FooService")
> protocol MyProtocol {
>
>     @namespace("FooServiceType")
>     record ObjFSPtr {
>         string uri;
>     }
>
>     string hash(FooServiceType.ObjFSPtr objfsptr, int x=10);
> }
>
>
> generated service.avpr:
>
> {
>   "protocol" : "MyProtocol",
>   "namespace" : "FooService",
>   "types" : [ {
>     "type" : "record",
>     "name" : "ObjFSPtr",
>     "namespace" : "FooServiceType",
>     "fields" : [ {
>       "name" : "uri",
>       "type" : "string"
>     } ]
>   } ],
>   "messages" : {
>     "hash" : {
>       "request" : [ {
>         "name" : "objfsptr",
>         "type" : "FooServiceType.ObjFSPtr"
>       }, {
>         "name" : "x",
>         "type" : "int",
>         "default" : 10
>       } ],
>       "response" : "string"
>     }
>   }
> }
>
>
> Thanks!
>
> Jonathan Camp

Mime
View raw message