atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barbara Eckman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ATLAS-2694) Avro schema typedef and support for Avro schema evolution in Atlas
Date Tue, 29 May 2018 15:38:00 GMT

     [ https://issues.apache.org/jira/browse/ATLAS-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Barbara Eckman updated ATLAS-2694:
----------------------------------
    Description: 
Currently the base types in Atlas do not include Avro schemas. It would be nice to add typedef
for Avro schema and any associated metadata to support schema evolution.
 * For example, Avro_schema type supports:
 ** All avro types, both primitive and complex, including union types, as fields of schema
 ** All types have doc strings and defaults
 ** A field of a schema can be another schema
 ** Indefinite nesting of records, arrays.
 ** Associated entities array attribute contains pointers to all datasets that reflect the
avro schema
 ** Fully expanded avroNotation for use in serDe
 ** Schema evolution features such as isLatest (Boolean) and version number
 * Schema evolution Process
 ** Input: avro schema
 ** Output: new version of avro schema
 ** Compatibility: FULL, BACKWARD, FORWARD, NONE
 ** IsBreakingChange (Boolean): does the change produce an incompatible schema? (ie its compatibility
is not “FULL”)
 *

  was:
Currently the base types in Atlas do not include Avro schemas. It would be nice to add typedef
for Avro schema and any associated metadata to support schema evolution.
 * For example, Avro_schema type supports:
 ** All avro types, both primitive and complex, including union types, as fields of schema
 ** All types have doc strings and defaults
 ** A field of a schema can be another schema
 ** Indefinite nesting of records, arrays.
 ** Associated entities array attribute contains pointers to all datasets that reflect the
avro schema
 ** Fully expanded avroNotation for use in serDe
 ** Schema evolution features such as isLatest (Boolean) and version number
 * Schema evolution Process
 ** Input: avro schema
 ** Output: new version of avro schema
 ** Compatibility: FULL, BACKWARD, FORWARD, NONE
 ** IsBreakingChange (Boolean): does the change produce an incompatible schema? (ie its compatibility
is not “FULL”)
 * Atlas UI enhancement for JSON-valued attributes to support avro schema and avro schema
evolution
 ** Currently JSON-valued attributes are fully displayed in-line with other attributes, not
pretty-printed, cluttering the display.  To support a better display, we can display JSON-valued
attributes in a one-line box that can be scrolled down, or fully expanded with a mouse click
that pretty-prints the JSON. 


Moving UI enhancement for avro schemas to a separate Jira 

> Avro schema typedef and support for Avro schema evolution  in Atlas
> -------------------------------------------------------------------
>
>                 Key: ATLAS-2694
>                 URL: https://issues.apache.org/jira/browse/ATLAS-2694
>             Project: Atlas
>          Issue Type: New Feature
>          Components:  atlas-core
>            Reporter: Srikanth Venkat
>            Assignee: Barbara Eckman
>            Priority: Critical
>             Fix For: 1.0.0
>
>         Attachments: ATLAS-2694-2.patch, ATLAS-2694.patch, avro_atlas_types_08.json
>
>
> Currently the base types in Atlas do not include Avro schemas. It would be nice to add
typedef for Avro schema and any associated metadata to support schema evolution.
>  * For example, Avro_schema type supports:
>  ** All avro types, both primitive and complex, including union types, as fields of schema
>  ** All types have doc strings and defaults
>  ** A field of a schema can be another schema
>  ** Indefinite nesting of records, arrays.
>  ** Associated entities array attribute contains pointers to all datasets that reflect
the avro schema
>  ** Fully expanded avroNotation for use in serDe
>  ** Schema evolution features such as isLatest (Boolean) and version number
>  * Schema evolution Process
>  ** Input: avro schema
>  ** Output: new version of avro schema
>  ** Compatibility: FULL, BACKWARD, FORWARD, NONE
>  ** IsBreakingChange (Boolean): does the change produce an incompatible schema? (ie its
compatibility is not “FULL”)
>  *



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message