atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <ames...@hortonworks.com>
Subject Review Request 59722: Import API: Support for resuming Import operation
Date Fri, 02 Jun 2017 20:52:17 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59722/
-----------------------------------------------------------

Review request for atlas and Madhan Neethiraj.


Bugs: ATLAS-1851
    https://issues.apache.org/jira/browse/ATLAS-1851


Repository: atlas


Description
-------

**Implementation**
- Added additional options to _AtlasImportRequest_.
  - Additional options:
    - _startGuid_
    - _startPosition_
- Added method for percentage calculation to _AtlasEntityStoreV1_.
- Updated logging message to include entity guid, type and position.

**CURL**

Create file with these contents call it _importTransform.json_:
```javascript
{ "options": {
    "startGuid": "bd97c78e-3fa5-4f9c-9f48-3683ca3d1fb1"
    }
}
```javascript


```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache"
-F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```

Steps to use the behavior:
- Start an import (using the CURL above) that is fairly long, say about 1000+ entities. 
- While the import is in progress, stop atlas server (using _atlas_stop.py_).
- From the log file located at _/var/log/atlas/application.log_ get the last successfully
imported entity GUID or position.
- Update the _importTransform.json_ with the guid.
- Restart import.

You should see that import resumes from where it left off.


**Highlights**
Specify the _startGuid_ option and notice that the operation resumes from the correct percentage
and not 0%.


Diffs
-----

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 4f2c1fbc 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 8a7e3585

  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 76451c98 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
27c0b5d4 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java
69140e69 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java
0f711db4 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1BulkImportPercentTest.java
PRE-CREATION 


Diff: https://reviews.apache.org/r/59722/diff/1/


Testing
-------

**Unit tests**
Added tests to cover the new functionality. Note the usage of mock for _Logger_.

**Volume tests**
- Performed large imports with resume.
- Noted the numbers against baseline. Did not observe significant deviation.

**Functional tests**
- Used common scenarios from test suite.

**Accuracy testing**
- Not done.


Thanks,

Ashutosh Mestry


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message