atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sarath Kumar Subramanian <sarath.ku...@gmail.com>
Subject Re: Review Request 51939: Framework to apply updates to types in the type-system
Date Wed, 21 Sep 2016 07:21:45 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
-----------------------------------------------------------

(Updated Sept. 21, 2016, 7:21 a.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
-------

added support to update and delete attributes in addition to add, addressed suma's code review
suggestions.


Bugs: ATLAS-1174
    https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
-------

1. Introduce "version" attribute to all types in the type-system, this helps to track changes
made to the default types (hive, sqoop, falcon and storm types) and user created types. If
version is not mentioned during creation of a type, default version "1.0" is assigned (optional
attribute).
2. Using the version attributed for types, introduce a patch framework for type system. This
framework applies patches to a type using its version number and can be used during upgrade
- add new attributes to an existing types and it will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any available patch
files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", "isComposite": false,
"isUnique": false, "isIndexable": false, "reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number and after
applying the patch, update the version to the one mentioned in "updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in the sort order
of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-----

  ATLAS-1171.7.patch PRE-CREATION 
  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
a94d157 
  repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java PRE-CREATION

  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 6a937f4 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java fad091d

  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java bdd0a13 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java 6340615

  typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java 7224699

  typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 6f40c1d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java f1ce1b7

  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 70ba89b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeUtils.java f5c2ce9 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java ef8448d

  typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 5618938


Diff: https://reviews.apache.org/r/51939/diff/


Testing
-------


Thanks,

Sarath Kumar Subramanian


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message