atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Radley (JIRA)" <>
Subject [jira] [Commented] (ATLAS-1690) Introduce top level relationships
Date Fri, 12 May 2017 10:30:04 GMT


David Radley commented on ATLAS-1690:

Hi [~madhan.neethiraj] and [~cmgrote],
This is interesting, it seems to me that if we are going to embrace using tags in context
specific ways and we feel tag propagation decisions should not be made by the relationship
author then we do not want to hard bake tag propagation into the Atlas store. I am hearing
that you feel there are use cases where the relationship author or updater would be responsible
for tag propagation.

Can I just check how you are thinking about the tag propagation implementation in your scenarios

1) when a table/column is classified as PII, any lineage from this table/view/column should
also be automatically be classified as PII. This means we need a relationship between an entity
defined attribute to another entity defined attribute in the other entity. The current proposal
does not allow this as the relationship specifies attribute names for each end, that are not
defined in the entity. The current relationship proposal adds a new attribute of type 'the
other end'. I think we would need a new tag propagating relationship - or more generally a

2) when a term is classified as PII, all entities that are associated with the term also should
automatically be classified as PII. So I am thinking that an asset / entity will have an assigned
terms attribute. The GlossaryTerm would have a assignedEntities attribute. These attributes
would be added by virtue of the relationship. If the Glossary Term was tagged PII, we need
special Atlas logic so assignedEntities propagates to the entities assignedTerms - then special
logic for the assignedTerms tag is picked up by the entity itself. 

3) when a term is classified as PII, all terms that are synonyms of this term (and all the
entities associated with the synonym terms) also should automatically be classified the same.
We would need special Atlas logic for the synonym case that propagates the PII tag from the
synonym relationship to the terms themselves.

In summary I see 2 main tag propagation scenarios, and propose a way forward for each:
- mapping between existing entity attributes, that could be used as the basis of propagation.
I suggest we introduce the top level concept of a mapping separately from this relationships
- the need for additional logic in Atlas to propagate tags across glossary term to term or
term to asset relationships. I can see this is useful. So I suggest that I add in tag_propagation_hint
enum including NONE. The actual propagation will depend on specific relationship types. When
we implement the glossary, we will use this hint to propagate the tags in glossary specific
Atlas logic. Further enhancements can occur as we bring in logic around the classification
level use case and enhanced Ranger integration; Ranger will need to be able to override the
hinted classification. Outside of the glossary case - tags will not be propagated - even if
the tag-propagation-hint is set on a relationship. Give this thinking,  it makes sense for
the tag_propagation_hint to be on only on the Glossary Relationship types and not on the top
level relationships.

Madhan and [~mandy_chessell] does this make sense? 


> Introduce top level relationships
> ---------------------------------
>                 Key: ATLAS-1690
>                 URL:
>             Project: Atlas
>          Issue Type: Improvement
>            Reporter: David Radley
>            Assignee: David Radley
>              Labels: VirtualDataConnector
>         Attachments: Atlas_RelationDef_Json_Structure_v1.pdf, Atlas Relationships proposal
v1.0.pdf, Atlas Relationships proposal v1.1.pdf, Atlas Relationships proposal v1.2.pdf, Atlas
Relationships proposal v1.3.pdf, Atlas Relationships proposal v1.4.pdf, Atlas Relationships
proposal v1.5.pdf, Atlas Relationships proposal v1.6.pdf, Atlas Relationships proposal v1.7.pdf
> Introduce top level relationships including support for 
> -many to many relationships
> - relationship names including the name for both ends and the relationship.

This message was sent by Atlassian JIRA

View raw message