atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <ames...@hortonworks.com>
Subject Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
Date Mon, 21 May 2018 02:15:53 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------

(Updated May 21, 2018, 2:15 a.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.


Changes
-------

Updates include: 
- Optimization for post processing.
- Refactoring of _RelationshipTypeCache_.
- Addressed review comments.


Bugs: ATLAS-2637
    https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
-------

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns
map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential
exeception when a non-existent property is checked for presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-----

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664

  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
c0b9c1741 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
16aecd5e2 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
ae119b0bc 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
ec320b03e 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
d0a65f7b1 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
e4e82649b 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
PRE-CREATION 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
e863d9fae 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
4d73c78ef 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
794b5471e 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
cac09d229 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
22cd55217 
  repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
3c84e3c22 
  repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38

  repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b

  repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
ec6e64a25 
  repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java
d810a1166 
  repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION 
  repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION

  repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 


Diff: https://reviews.apache.org/r/66928/diff/18/

Changes: https://reviews.apache.org/r/66928/diff/17-18/


Testing
-------

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.

**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)**


Thanks,

Ashutosh Mestry


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message