atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nigel Jones (JIRA)" <>
Subject [jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model
Date Thu, 20 Jul 2017 12:36:00 GMT


Nigel Jones commented on ATLAS-1839:

[~davidrad] the toxic combination support in ranger policies is primarily geared to controlling
what a user may access, whilst the validation [~ivarea] is suggesting is primarily about creates
and updates, ie defining the data model itself. That's not to say ranger couldn't do this
(since it can address any operation such as a create) but I don't think that's ranger's intent.
But I agree it's a fine line and could well vary significantly in different environments

As such I think it makes sense to define validation in atlas and be able to link to code artifacts,
services that implement those validations probably through a combination of discovery &
stewardship , plus making it easier when writing pipelines for say ETL or streaming, to be
able to easily pull in atlas metadata and capture a link between a validation implemented
by an pipeline author (or being used from a library) and it's definition in atlas. Thus atlas
ends up with both the "intent" (the business spec if you like) as well as links to the implementation
yet does not constrain those implementations since they can be so varied. 

Following on from this, absolutely some of those validations could be implemented as complex
rules, but I think it would be tricky and constraining to capture all that in atlas, hence
why I'd go for the link approach & some relatively loose coupling

So with that done, sure we could have a more complex rules engine embedded in, or used by
ranger plugins... but this could be one of a number of different approaches

I'd be inclined to start off with us figuring out how to model, and some use cases where we
can explore the authoring (ie in atlas), assisted authoring (when writing a job), metadata
capture (from those other systems, also relates to lineage) & probably best to do that
in ATLAS-1995? This also touches on RANGER-1869 (metadata capture)

Certainly this is an interesting area !

> Area 2 of the open metadata model
> ---------------------------------
>                 Key: ATLAS-1839
>                 URL:
>             Project: Atlas
>          Issue Type: Task
>          Components:  atlas-core
>    Affects Versions: 0.9-incubating
>            Reporter: Mandy Chessell
>            Assignee: David Radley
>              Labels: OpenMetadata, VirtualDataConnector
>         Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 0220CategoryHierarchy.json,
0230Terms.json, 0240Dictionary.json, 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json,
> This task delivers the JSON files for the new models that describe types for Area 2 in
the open metadata model. This area covers the glossary.

This message was sent by Atlassian JIRA

View raw message