atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <ames...@hortonworks.com>
Subject Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids
Date Sun, 20 May 2018 16:27:36 GMT


> On May 19, 2018, 12:36 a.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
> > Line 67 (original), 74 (patched)
> > <https://reviews.apache.org/r/66928/diff/16/?file=2025013#file2025013line74>
> >
> >     This one handles removal/updte of properties for map type attributes. Aren't
similar updates needed for array type attributes as well (primitive/struct/entity-ref)?

Yes, only that the indexes are known only after all edges are created. I considered doing
this during creation itself, it eded up becoming messy and number of computations creases
drastically. There is an option that is possible if it is some how possible to store a temporary
map of edge ids to indexes.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203462
-----------------------------------------------------------


On May 17, 2018, 8:48 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> -----------------------------------------------------------
> 
> (Updated May 17, 2018, 8:48 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2637
>     https://issues.apache.org/jira/browse/ATLAS-2637
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_,
returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it down to
migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential
exeception when a non-existent property is checked for presence.
> 
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
> 
> 
> Diffs
> -----
> 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664

>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
c0b9c1741 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
16aecd5e2 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
ae119b0bc 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
ec320b03e 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
d0a65f7b1 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
e4e82649b 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
PRE-CREATION 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
e863d9fae 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
4d73c78ef 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
794b5471e 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
cac09d229 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
PRE-CREATION 
>   graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
>   graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
>   repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
22cd55217 
>   repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
3c84e3c22 
>   repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
ac0b79d38 
>   repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
ffbf3200b 
>   repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
ec6e64a25 
>   repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java
PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java
d810a1166 
>   repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION

>   repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION

>   repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 
> 
> 
> Diff: https://reviews.apache.org/r/66928/diff/16/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> 
> Additional tests added.
> 
> **Functional tests**
> 
> Regular flow verified.
> 
> **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/416/)**
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message