atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <ames...@hortonworks.com>
Subject Re: Review Request 59722: Import API: Support for resuming Import operation
Date Sat, 03 Jun 2017 15:32:58 GMT


> On June 2, 2017, 9:28 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
> > Lines 208 (patched)
> > <https://reviews.apache.org/r/59722/diff/1/?file=1740869#file1740869line218>
> >
> >     Why 'currentIndex + 1'? ZipSource.currentIndex seems to have the number of entities
processed so far. Please review. Also 'ZipSource.currentPosition' might be a better name,
instead of currentIndex. Please review.

There small accuracy problem that comes up when starting with 0 to calculate %age. Starting
with 1 seems to handle all cases well.


> On June 2, 2017, 9:28 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java
> > Lines 48 (patched)
> > <https://reviews.apache.org/r/59722/diff/1/?file=1740870#file1740870line48>
> >
> >     Wouldn't size be always "1"? Given this object is initialized with an instance
of AtlasEntityWithExtInfo?

Intent here is to discourage use of this class. Your observation is valid. I will change it.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59722/#review176849
-----------------------------------------------------------


On June 2, 2017, 10:18 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59722/
> -----------------------------------------------------------
> 
> (Updated June 2, 2017, 10:18 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1851
>     https://issues.apache.org/jira/browse/ATLAS-1851
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Implementation**
> - Added additional options to _AtlasImportRequest_.
>   - Additional options:
>     - _startGuid_
>     - _startPosition_
> - Added method for percentage calculation to _AtlasEntityStoreV1_.
> - Updated logging message to include entity guid, type and position.
> 
> **CURL**
> 
> Create file with these contents call it _importTransform.json_:
> ```javascript
> { "options": {
>     "startGuid": "bd97c78e-3fa5-4f9c-9f48-3683ca3d1fb1"
>     }
> }
> ```javascript
> 
> 
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control:
no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
> ```
> 
> Steps to use the behavior:
> - Start an import (using the CURL above) that is fairly long, say about 1000+ entities.

> - While the import is in progress, stop atlas server (using _atlas_stop.py_).
> - From the log file located at _/var/log/atlas/application.log_ get the last successfully
imported entity GUID or position.
> - Update the _importTransform.json_ with the guid.
> - Restart import.
> 
> You should see that import resumes from where it left off.
> 
> 
> **Highlights**
> Specify the _startGuid_ option and notice that the operation resumes from the correct
percentage and not 0%.
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 4f2c1fbc 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 8a7e3585

>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 76451c98

>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
27c0b5d4 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java
69140e69 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java
0f711db4 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1BulkImportPercentTest.java
PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59722/diff/2/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Added tests to cover the new functionality. Note the usage of mock for _Logger_.
> 
> **Volume tests**
> - Performed large imports with resume.
> - Noted the numbers against baseline. Did not observe significant deviation.
> 
> **Functional tests**
> - Used common scenarios from test suite.
> 
> **Accuracy testing**
> - Not done.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message