atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sarath Kumar Subramanian <sarath.ku...@gmail.com>
Subject Re: Review Request 51939: Framework to apply updates to types in the type-system
Date Tue, 20 Sep 2016 17:50:29 GMT


> On Sept. 20, 2016, 1:11 p.m., David Radley wrote:
> > It looks like this is a way of updating the versions of types using a local file
on the Atlas server. I think we should be SaaS and expose this sort of functionality primarily
as a REST API, otherwise this could become a barrier to integration.

David, the goal of this framework is to apply patches during upgrade scenarios - when user
has changed the shape of a type (added new attributes, new supertypes) or to update an existing
type. The patch files will be checked and applied during atlas startup, this will avoid dealing
with multiple patch scripts for upgrading multiple types. Currrently a type can be updated
using REST (PUT - /api/atlas/types) by including the type definition in request payload. We
can add only optional attributes, so instances depending on old type will not be broken.

I can add a dedicated REST layer to this once we have ATLAS-1171 ready.


- Sarath Kumar


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
-----------------------------------------------------------


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> -----------------------------------------------------------
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
>     https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps to track
changes made to the default types (hive, sqoop, falcon and storm types) and user created types.
If version is not mentioned during creation of a type, default version "1.0" is assigned (optional
attribute).
> 2. Using the version attributed for types, introduce a patch framework for type system.
This framework applies patches to a type using its version number and can be used during upgrade
- add new attributes to an existing types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any available
patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", "isComposite":
false, "isUnique": false, "isIndexable": false, "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version number and after
applying the patch, update the version to the one mentioned in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in the sort
order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java PRE-CREATION

>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java PRE-CREATION

>   repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 6a937f4

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java fad091d

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java bdd0a13 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
6340615 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java 7224699

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 6f40c1d

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 70ba89b

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java ef8448d

>   typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message