atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sarath Subramanian <sar...@apache.org>
Subject Re: Review Request 65700: [ATLAS-2456]: Implement tag propagation using relationships
Date Sun, 18 Feb 2018 18:25:04 GMT


> On Feb. 18, 2018, 2:18 a.m., David Radley wrote:
> > addons/models/0000-Area0/0010-base_model.json
> > Line 216 (original), 216 (patched)
> > <https://reviews.apache.org/r/65700/diff/2/?file=1962389#file1962389line216>
> >
> >     you are adding tag propagation from a DataSet to a Process and from a Process
to a dataset. I realise these are aggregation relationships, I wonder what prevents circularities.

We handle this in 2 places one in the tp3 gremlin query usign simplePath() (that avoids circular
loops) and also in the server side code that checks if the classification is not already propagated
(assigned).


> On Feb. 18, 2018, 2:18 a.m., David Radley wrote:
> > common/src/main/java/org/apache/atlas/repository/Constants.java
> > Lines 72 (patched)
> > <https://reviews.apache.org/r/65700/diff/2/?file=1962390#file1962390line72>
> >
> >     Can we use classification instead of traits or tags?

At the vertex level, we still use v1 names. Entity classification names are mentioned in a
list - 'traitNames', to keep it consistent I used 'propagatedTraitNames'


> On Feb. 18, 2018, 2:18 a.m., David Radley wrote:
> > dashboardv2/public/js/utils/Enums.js
> > Lines 31 (patched)
> > <https://reviews.apache.org/r/65700/diff/2/?file=1962391#file1962391line31>
> >
> >     agree with the previous reviewer that these tags seem the same as the TAG_ enums

will remove it


> On Feb. 18, 2018, 2:18 a.m., David Radley wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
> > Lines 1141 (patched)
> > <https://reviews.apache.org/r/65700/diff/2/?file=1962418#file1962418line1148>
> >
> >     I wonder if you could explain what ths creation of a propagation edge means.
This implies that the propagated classifications are stored in the edges. I am wondering how
this fits with associating the propagated classifications with the downstream entities, which
I assume will be stored as propagation classifications in the downstream entities - have /i
understood this correctly. A comment to explain how this works would be helpful.

When an entityA is assigned with classification 'PII', all its impacted/downstream entity
vertices (B, C and D) will now have a 'propagated edge' with edge label - propagated:PII to
the classification vertex, also the classification name is added to the propagatedTraitNames
property of the impacted vertices (for search purpose). I have detailed this approach in the
design doc attached to https://issues.apache.org/jira/browse/ATLAS-2456


- Sarath


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65700/#review197732
-----------------------------------------------------------


On Feb. 18, 2018, 10:10 a.m., Sarath Subramanian wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65700/
> -----------------------------------------------------------
> 
> (Updated Feb. 18, 2018, 10:10 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2456
>     https://issues.apache.org/jira/browse/ATLAS-2456
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Scalable way to quickly and efficiently propagate tags for efficient searches and tag
based security. Likewise tags for derivative dataset should be inherited from the parent.
For example, if an entity is tagged "PII" then resulting entity created from a CTAS operation
should also be tagged "secret" to maintain the classification of the parent. In the case where
2 or more datasets are aggregated the derivative dataset should be a union of all parent tags.
> 
> This changeset includes introduction of v2 data structures for notifications and audit
events.
> 
> 
> Diffs
> -----
> 
>   addons/models/0000-Area0/0010-base_model.json 0296e8f9 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 265be788 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 42837f49

>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java a68d8ebe

>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java
aef20f03 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
ca48e3d9 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java ff09e6c9 
>   intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java PRE-CREATION

>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java PRE-CREATION

>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java f594a814

>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 08d1ce11 
>   intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java b272b733

>   intg/src/main/java/org/apache/atlas/v1/model/notification/EntityNotificationV2.java
PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
74d3b913 
>   repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
9dc78350 
>   repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
774934c7 
>   repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
22d2a810 
>   repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
c3826019 
>   repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
2884f8f0 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
31620b1e 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java d61bff24

>   repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
79e8e3e8 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityChangeNotifier.java
2b6beade 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
ca0eeeb6 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
7389f49b 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
779bc387 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
b05a9a32 
>   repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 1fda2415

>   repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java a79abaaf

>   repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java 0e1e5b6c

>   repository/src/test/java/org/apache/atlas/TestModules.java 13bdcb08 
>   repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
47a61ee2 
>   webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
396a2920 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 715a54db 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 21402e19 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java ea6fe313 
>   webapp/src/test/resources/atlas-application.properties d9009166 
> 
> 
> Diff: https://reviews.apache.org/r/65700/diff/4/
> 
> 
> Testing
> -------
> 
> mvn clean install -> succeeded in 17:11 min
> 
> UTs/ITs for tag propagation work in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message