atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <>
Subject Review Request 66184: Migration Utility: Branch 0.8: Performance Improvement
Date Tue, 20 Mar 2018 23:14:01 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.

Bugs: ATLAS-2461

Repository: atlas


The migration utility committed ealier has couple of short comings:
- Relies on Export service.
  - Needs _export-options.json_ to be specified.
  - Exporting everything means meticuloulsy updating the options file. It is likely some specification
is missed and hence will lead to less data being migrated. 
- Suffers from performance problems for large data sets.

The new approach uses _Titan's_ _GraphSON_ writer. This is configured to export all data in
_EXTENDED_ format.

The _EXTENDED_ format separates _vertices_ and _edges_. This open other interesting avenues
for import.

- Modified _Exporter_ to use _AtlasTypeRegistry_ and _GraphSONWriter_.
- Produced files: 
   - _atlas-typedef.json_: Contains type definitions of all types.
   - _atlas-migration-data.json_: Contains data from the database.


  tools/atlas-migration-exporter/pom.xml 5c6c61ee 
  tools/atlas-migration-exporter/src/main/java/org/apache/atlas/migration/ a9873df0



**Functional tests**
Export from repositories with:
- Custom types.
- Complex lineages.
- Created hive entities via beeline.
- Imported data.

**Gremlin Shell**
- Used _Gremlin_ shell to perform export operation.


Ashutosh Mestry

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message