Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@atlas.incubator.apache.org
Date: Tue, 9 May 2017 08:13:04 +0000 (UTC)
From: "David Radley (JIRA)" <jira@apache.org>
To: dev@atlas.incubator.apache.org
Message-ID: <JIRA.13060159.1490821989000.161652.1494317584576@Atlassian.JIRA>
In-Reply-To: <JIRA.13060159.1490821989000@Atlassian.JIRA>
References: <JIRA.13060159.1490821989000@Atlassian.JIRA> <JIRA.13060159.1490821989308@jira-lw-us.apache.org>
Subject: [jira] [Commented] (ATLAS-1690) Introduce top level relationships
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 09 May 2017 08:13:15 -0000


    [ https://issues.apache.org/jira/browse/ATLAS-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002261#comment-16002261 ] 

David Radley commented on ATLAS-1690:
-------------------------------------

Hi [~madhan.neethiraj] 
Great feedback - thanks for your thoughtful and open response.
I will change the array to and endpoint1 and 2. I think that is clearer.

I am keen that we propagate tags, this is very powerful. 
I thought I would explain how we could do classifications and then see how this option fits in.
The classifications our customers are working with include confidentiality. The confidentiality scheme might have C1,C2, C3 and C4 levels. C1 might be public and C3 top secret. Different companies name these Cn levels differently. But in all these cases there is an order, C4 being the highest classification level. Though it is possible to have more complex classifications schemes- many / most cases can work with this list sort of classification schema. A particular glossary term or column might be associated with one of these classifications. 
We were thinking that the Classification levels (C1, C2 etc) would be new system Types (I suggest an EntityDef) A classification use is the relationship between the classification level and the thing it is classifying. By default the classification will be that from the level , but a rule can run and increase the classification; this would be calculated (and stored?) in the relationship instance.      

So to address your proposal:
- I think the propagated classifications would be derived at query time and could be useful -do we need an effective classification? I am not convinced with the proposed mechanism.
- Your example around tables and columns and PII assumes that PII is a binary flag (or one tag), I am suggesting that this is not the way that classifications are normally implemented - these should be an ordered list of levels. I see in some of your recent demos you use v1 terms to implement these classification levels for this classification ordering. If a table is classified as public and has a PII column , we would not want the public classification to override the PII column.  As a query brings together 2 public fields, line name and salary, the combination becomes PII, in this case we need the rule to drive this.
This implementation would encourage the use of bidirectional relationships to be implemented purely to propagate tags. I suggest many propagations would not be on one relationship, but could flow much further would be to all has-a terms - following all the has-a links. 
I am also concerned that the role who authors the relationship is not the right role to make classification propagation decisions.  

I wonder whether a smarter approach would be to tag the relationship as "propagate-1-to-2" (hopefully something more meaningful like propogate-table-to-column")  and Ranger picks up this hint. Ranger could decide to run a simple rule of propagating all the tags from 1 to 2 or a more complex rule taking other conditions into account.

I suggest that we explicitly implement  these classification levels and uses, I hope there is a simple case where there are some classifications that should be propagated for all consumer cases, and rules can run to override the classifications and we can find a way of doing this using a governance role and we could make this work. Maybe some supplied Ranger rules and tags that Atlas used out of the box. GDPR rules and tags would be a good use case here.


> Introduce top level relationships
> ---------------------------------
>
>                 Key: ATLAS-1690
>                 URL: https://issues.apache.org/jira/browse/ATLAS-1690
>             Project: Atlas
>          Issue Type: Improvement
>            Reporter: David Radley
>            Assignee: David Radley
>              Labels: VirtualDataConnector
>         Attachments: Atlas_RelationDef_Json_Structure_v1.pdf, Atlas Relationships proposal v1.0.pdf, Atlas Relationships proposal v1.1.pdf, Atlas Relationships proposal v1.2.pdf, Atlas Relationships proposal v1.3.pdf, Atlas Relationships proposal v1.4.pdf, Atlas Relationships proposal v1.5.pdf, Atlas Relationships proposal v1.6.pdf, Atlas Relationships proposal v1.7.pdf
>
>
> Introduce top level relationships including support for 
> -many to many relationships
> - relationship names including the name for both ends and the relationship.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)