avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John E. Conlon" <jcon...@verticon.com>
Subject Re: Can I have metadata per field in Avro?
Date Tue, 16 May 2017 01:50:32 GMT
Hi Kant,

I use the doc values to store additional metadata for the record and the 
individual fields so my various consuming components can know what to do 
with the Avro records.

For the record I have components that index Avro records to 
Elasticsearch so I use the token:
@_index=[events-identity]
to indicate the Elasticsearch index name.

For fields within the record I use tokens like:
@_id
@_mapped:com.verticon.im.avro.TagReportEvent.epc
the first to indicates the field that to be used as an id(or key) and 
the second to indicate that this field was mapped from another field 
from another Avro record.

Of course the @_ was just my way of indicating embedded metadata you 
could use whatever you want.

John


On 05/15/2017 08:21 PM, Artem Ervits wrote:
> What you're asking for is better suited for a governance and metadata 
> management solution. Apache Atlas and Cloudera Navigator are in that 
> realm though I am not intimately familiar with the latter. With Atlas 
> you can maintain additional metadata and track data lineage.
>
> On May 15, 2017 9:14 PM, "kant kodali" <kanth909@gmail.com 
> <mailto:kanth909@gmail.com>> wrote:
>
>     Hi All,
>
>     Can I store some business metadata per field? other words, I have
>     some business metadata I want to specify per field is that
>     possible in Avro? Any examples
>
>     Thanks!
>



Mime
View raw message