atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Radley <david_rad...@uk.ibm.com>
Subject Re: Review Request 51939: Framework to apply updates to types in the type-system
Date Fri, 16 Sep 2016 08:56:17 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149181
-----------------------------------------------------------



I like the idea of introducing versions for types. I would like it to be clear what the use
cases are for this.

I do not fully understand the fix and have not applied it yet - apologises if I have misunderstood
the context. I wonder :
1) when do you envisage versioning a default type vs subclassing it. It seems to me subclassing
is more flexible in many use cases. 
2) Changing the shape of a type will effect any objects that depend on the old type shape
- so all the instance data in Atlas and data in the users application that depends on the
old shape amy break.  
3) What would a potential difference be between one version of a type and a higher version
- what can you add, can you only add at the end of an object , what can you remove, how does
this effect any children types? Is this purely for patch management or also exposed at runtime
in the APIs.
4) This reminds me of previous thinking we have used. We created a system where we had types;
you could only add attributes and adding optional attributes would not require a new version.
Only when new mandatory attributes were added would we require a new version. Is this pattern
relevant here?
4) Are we expecting versions to basically get bigger and better in a chain or are we expecting
splits. If we are a chain, I would suggest there are use cases where creation of a new entity
based on a type picks up the latest rather than the default. So example all the entities in
a project.  
5) If we update a version in a type, presumable we need to keep the old version around so
we can process entities and the like that depend on that shape. The  updateToVersion implies
that we have lost the old version - presumable existing data instances referring to the old
type still need to work in queries and the like; presumably we are doing a soft delete to
allow lineage to occur.

David Radley

- David Radley


On Sept. 15, 2016, 11:03 p.m., Sarath Kumar Subramanian wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> -----------------------------------------------------------
> 
> (Updated Sept. 15, 2016, 11:03 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
>     https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps to track
changes made to the default types (hive, sqoop, falcon and storm types) and user created types.
If version is not mentioned during creation of a type, default version "1.0" is assigned (optional
attribute).
> 2. Using the version attributed for types, introduce a patch framework for type system.
This framework applies patches to a type using its version number and can be used during upgrade
- add new attributes to an existing types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any available
patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", "isComposite":
false, "isUnique": false, "isIndexable": false, "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version number and after
applying the patch, update the version to the one mentioned in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in the sort
order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java PRE-CREATION

>   repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 6a937f4

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java fad091d

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java bdd0a13 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
6340615 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java 7224699

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 6f40c1d

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 70ba89b

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java ef8448d

>   typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message