Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@atlas.incubator.apache.org
Content-Type: multipart/alternative;
 boundary="===============5070774115633539494=="
MIME-Version: 1.0
Subject: Re: Review Request 51939: Framework to apply updates to types in the
 type-system
From: David Radley <david_radley@uk.ibm.com>
To: Madhan Neethiraj <madhan@apache.org>,
 Shwetha GS <sshivalingamurthy@hortonworks.com>
Cc: David Radley <david_radley@uk.ibm.com>,
 atlas <dev@atlas.incubator.apache.org>,
 Sarath Kumar Subramanian <sarath.kum4r@gmail.com>
Date: Tue, 20 Sep 2016 13:11:41 -0000
Message-ID: <20160920131141.1654.47942@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: David Radley <noreply@reviews.apache.org>
References: <20160919181056.25295.82847@reviews.apache.org>
In-Reply-To: <20160919181056.25295.82847@reviews.apache.org>
Reply-To: David Radley <david_radley@uk.ibm.com>
archived-at: Tue, 20 Sep 2016 13:11:48 -0000

--===============5070774115633539494==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
-----------------------------------------------------------


It looks like this is a way of updating the versions of types using a local file on the Atlas server. I think we should be SaaS and expose this sort of functionality primarily as a REST API, otherwise this could become a barrier to integration.

- David Radley


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> -----------------------------------------------------------
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
>     https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps to track changes made to the default types (hive, sqoop, falcon and storm types) and user created types. If version is not mentioned during creation of a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for type system. This framework applies patches to a type using its version number and can be used during upgrade - add new attributes to an existing types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", "isComposite": false, "isUnique": false, "isIndexable": false, "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version number and after applying the patch, update the version to the one mentioned in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 6a937f4 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java bdd0a13 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java 6340615 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java 7224699 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java 9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 6f40c1d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 70ba89b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java ef8448d 
>   typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>


--===============5070774115633539494==--