atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <ames...@hortonworks.com>
Subject Review Request 56795: Import REST API
Date Fri, 17 Feb 2017 19:57:19 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56795/
-----------------------------------------------------------

Review request for atlas, Madhan Neethiraj and Sarath Subramanian.


Bugs: ATLAS-1503
    https://issues.apache.org/jira/browse/ATLAS-1503


Repository: atlas


Description
-------

The patch includes the optimization done for import:
- Handling of large databases (151 tables, each with 80 columns)
- Improved handling of zip file inputs.
- Improved metrics data. This now includes granular breakdown of individual entities and list
of all entities imported.

Import is implemented in 2 flavors:
- ZIP file upload in POST.
   CURL:
   ```
   curl -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control:
no-cache" --data-binary @../docs/DefaultDB-Stocks.zip "http://localhost:21000/api/atlas/admin/import"
   ```
- Specify ZIP file as part of query parameter in GET.
   CURL:
   ```
   curl -X POST -u admin:admin -H "Cache-Control: no-cache" "http://localhost:21000/api/atlas/admin/importFile?FILENAME=/Users/amestry/Documents/DefaultDB-Sales.zip"
   ```


Diffs
-----

  intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 584bf25 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java a5eeef1 
  intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java 74e3c57 
  repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 1cd4375

  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
4312287 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStream.java
010b626 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
2e71ab8 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java c8c0099 
  webapp/src/main/java/org/apache/atlas/web/resources/ExportService.java 7d3d442 
  webapp/src/main/java/org/apache/atlas/web/resources/ImportService.java 7554cdb 
  webapp/src/main/java/org/apache/atlas/web/resources/ZipSource.java ea62862 

Diff: https://reviews.apache.org/r/56795/diff/


Testing
-------

Status:
- Large DB (151 tables): Export.
- - Quick start databases: Done.
  - Includes lineage information.
- hive_db database with upload from CSV file: Done.
- Unit tests: Pending.


Thanks,

Ashutosh Mestry


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message