avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Schmitt <dan.schm...@gmail.com>
Subject C++ interface, json, schema, immuatability
Date Thu, 06 Jul 2017 21:07:53 GMT
I was hoping I wasn't doing something complex, but I may be wrong.

My goal state is to allow callers to send arbitrary avro buffers to
me, and then merge them into a schema.

I seem to be stuck in that the exposed methods to extract the schema
from the buffer/stream etc result in a ValidSchema, which I don't seem
to be able to convert to something I can add to an existing schema.

As an example, imagine I have a caller passing me a hunk of memory
with data that has defined their own schema as:

  "type" : "record",
  "name" : "ErrorInfo",
  "fields" : [ {
    "name" : "File",
    "type" : "string"
  }, {
     "name" : "LineNumber",
     "type" : "int"
  } , {
    "name" : "Message",
    "type" : :string"
  } ]

And I want to store each hunk as a record with a timestamp when I got it.

  "type" : "record",
  "name" : "Log",
  "fields" : [ {
      "name" : "time",
      "type" : {
        "type" : "long",
        "logicalType" : "timestamp-millis"
      }, {
        "name" : "payload",
        "type" : [ "null" ]
        } ]

I'd like to be able to load up my timestamp log fields, then modify
the payload union to include the ErrorInfo type so somebody that dumps
the avro file can see File/LineNumber/Message and my logger doesn't
have to care about that bit of info (and anybody can make send any
type of structured data to the logger.)

I don't see a way to convert the node/root from ValidSchema to
something I can pass to avro::UnionSchema.addType()   Am I missing

Note that a timestamp logger is a contrived example to express what I
want to do (I know about all the nifty logging systems out there, but
I do have a use case for doing dynamic schema merges.)

View raw message