atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <ames...@hortonworks.com>
Subject Re: Review Request 65779: Data Migration: Moving Data to New Version of Atlas
Date Wed, 28 Feb 2018 21:20:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/
-----------------------------------------------------------

(Updated Feb. 28, 2018, 9:20 p.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.


Changes
-------

Updates include:
- Fixed place where notification sending is blocked.
- Updated location of _log.info_ message in _Services_.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists,
all services except _DataMigrationService_ is started. Migration is started. Atlas server
is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server
starts up.
- _ImportProgressTracker_: Added to be able to query progress of import.
- _ImportService_: Modified to send updates to _ImportProgressTracker_.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking
at the configuration parameter above.
- _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache"
http://localhost:21000/api/atlas/admin/status
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f 
  pom.xml 50476262 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION

  repository/src/main/java/org/apache/atlas/repository/impexp/DirectorySource.java PRE-CREATION

  repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION

  repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593

  repository/src/main/java/org/apache/atlas/repository/impexp/ImportSource.java PRE-CREATION

  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java ecfbd242 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityChangeNotifier.java
b9945d4f 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
b5461d4b 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java
af310954 
  repository/src/test/java/org/apache/atlas/repository/impexp/DirectorySourceTest.java PRE-CREATION

  repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 6fb6b454

  repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 080a96ac

  repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
d8e916d7 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
ac35860d 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/4e13b36b-9c54-4616-9001-1058221165d0.json PRE-CREATION

  repository/src/test/resources/stocks-2/5cfc2540-9947-40e0-8905-367e07481774.json PRE-CREATION

  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/65779/diff/8/

Changes: https://reviews.apache.org/r/65779/diff/7-8/


Testing
-------

**Unit tests**
Unit tests for _DirectorySource_ added.

**Functional tests**
Steps to test:
- Place the exported file say _export-1.zip_ at _/root_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/export-1.zip_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.


Thanks,

Ashutosh Mestry


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message