atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <ames...@hortonworks.com>
Subject Re: Review Request 67302: Data Migration: Handle Legacy Types that Reference Classifications
Date Fri, 25 May 2018 06:18:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67302/
-----------------------------------------------------------

(Updated May 25, 2018, 6:18 a.m.)


Review request for atlas, Madhan Neethiraj and Ruchi Solani.


Changes
-------

Updated: Added section _Refactoring_.


Bugs: ATLAS-2717
    https://issues.apache.org/jira/browse/ATLAS-2717


Repository: atlas


Description (updated)
-------

**Approach**
- Attributes of classification types are pre-processed (before incoming types are processed).
- New _STRUCT_ types are created with the classification name prefixed with _legacy_. So,
classification of the name PII is used to create a _STRUCT_ type named _legacy_PII_.
- Entity definitions are updated to reference these newly created types.
- During edge processing, the type name of the in vertex is replaced with this new type.
- Rest of the processing continues as is.

**Refacoring**
- Moved migration-specific functions to separate class _GraphDBMigrator_.


Diffs
-----

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java e5316d80a

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDBMigrator.java PRE-CREATION

  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
b4d6b33ea 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
c9d6067db 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java
f51080a61 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphDBGraphSONMigrator.java
PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesDefScrubber.java
PRE-CREATION 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
aee1b698c 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
049fd6418 
  graphdb/janus/src/test/resources/entity-with-trait-type.json PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java
9620c1357 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
081376ff5 
  repository/src/test/java/org/apache/atlas/TestModules.java 23d128740 
  repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
800638d43 
  repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java cae751215

  repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java 9f255189d

  repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
4b02b7fb7 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationProgressServiceTest.java
8fc524e81 
  repository/src/test/java/org/apache/atlas/repository/migration/PathTest.java 468b5268c 
  repository/src/test/java/org/apache/atlas/repository/migration/TypesDefScrubberTest.java
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/TypesWithClassificationTest.java
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java
4b236c1fe 
  repository/src/test/resources/classification_defs/atlas-migration-data.json PRE-CREATION

  repository/src/test/resources/classification_defs/atlas-migration-typesdef.json PRE-CREATION

  repository/src/test/resources/legacy-typesdef.json PRE-CREATION 


Diff: https://reviews.apache.org/r/67302/diff/3/


Testing
-------

**Unit tests**
- Added tests to verify the new flow.
- Executed existing tests.

**Test data**
- In versions 2.5 onwards (post 2/13/2017) it is not possible to create entities of such types.
Entity creation API throws a validation error.
- The test data was thus hand-created.
- This data verifies the cases where attribute type is:
  - Classification type
  - Array of classification type.
  - Map of classification type.

**Build**
[Pre-commit](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/451/)


File Attachments
----------------

atlas-migration-typesdef.json
  https://reviews.apache.org/media/uploaded/files/2018/05/24/c5cc2171-27f5-4d26-acd9-08aac9e1e1ef__atlas-migration-typesdef.json
atlas-migration-data.json
  https://reviews.apache.org/media/uploaded/files/2018/05/24/8230c44b-9f2e-4ae2-bc1a-04424dafc08c__atlas-migration-data.json


Thanks,

Ashutosh Mestry


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message