atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <>
Subject Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import
Date Thu, 25 May 2017 05:16:22 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated May 25, 2017, 5:16 a.m.)

Review request for atlas and Madhan Neethiraj.


Fixed problem with enum used as factory.

Bugs: ATLAS-1825

Repository: atlas


Please refer to [ATLAS-1825]( for scenarios
and solution approach.

* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of
entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string
of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
    * _replace_ Implements find and replace on attribute values.
    * _lowercase_ Converts attribute value to lower case. 
* Additional classes that unit test these classes have been added.

Implementation Approach
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity
that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters
as well as the binary input from the file.
* Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.


Create file with these contents call it _importTransform.json_:
    "options": {},
    "importTransform": {
        "transforms": {
            "hive_table": {
                "qualifiedName": [

curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache"
-F request=@../docs/importTransform.json -F data=@../docs/ "http://localhost:21000/api/atlas/admin/import"

Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.

Notice the output from the API, it has the _AtlasRequest_ that was passed as input.

Need to update TWIKI documents.

Diffs (updated)

  intg/src/main/java/org/apache/atlas/model/impexp/ 7530196d 
  intg/src/main/java/org/apache/atlas/model/impexp/ PRE-CREATION

  intg/src/main/java/org/apache/atlas/model/impexp/ PRE-CREATION

  intg/src/test/java/org/apache/atlas/model/impexp/ PRE-CREATION

  pom.xml a93b8ad9 
  repository/src/main/java/org/apache/atlas/repository/impexp/ 9ec15e00

  repository/src/main/java/org/apache/atlas/repository/impexp/ 87c9f0eb 
  repository/src/test/java/org/apache/atlas/repository/impexp/ 08e9ee81

  repository/src/test/java/org/apache/atlas/services/ d4900838

  webapp/pom.xml 4132912d 
  webapp/src/main/java/org/apache/atlas/web/resources/ 01f9a860 




**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.

**Functional Test**
* Small size data export & import.
* Medium size data export & import.

**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.


Ashutosh Mestry

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message