atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Dossett (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-409) Atlas will not import avro tables with schema read from a file
Date Wed, 23 Dec 2015 13:59:46 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15069645#comment-15069645
] 

Aaron Dossett commented on ATLAS-409:
-------------------------------------

[~shwethags] This is an interesting scenario.  With the schema defined in a file, it's possible
the schema in the file can change in allowed ways.  This can definitely happen with avro,
which allows schema changes.  If the bridge does read the columns through the schema file,
how would atlas keep up to date if the schema file changes?

> Atlas will not import avro tables with schema read from a file
> --------------------------------------------------------------
>
>                 Key: ATLAS-409
>                 URL: https://issues.apache.org/jira/browse/ATLAS-409
>             Project: Atlas
>          Issue Type: Bug
>    Affects Versions: 0.6-incubating
>            Reporter: Aaron Dossett
>            Assignee: Aaron Dossett
>         Attachments: ATLAS-409.patch, example.ddl, foo.avsc
>
>
> Atlas won't import a Hive table with no columns (see below for an example of a valid
hive table with no explicit columns).  This is because the Atlas Hive Storage Descriptor class
REQUIRES columns, but the Hive Table class allows them to be OPTIONAL.
> {code}
> CREATE TABLE example
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   TBLPROPERTIES (
>     'avro.schema.url'='file:///path/to/the/schema/test_serializer.avsc');
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message