atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <ames...@hortonworks.com>
Subject Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import
Date Thu, 25 May 2017 05:16:22 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------

(Updated May 25, 2017, 5:16 a.m.)


Review request for atlas and Madhan Neethiraj.


Changes
-------

Fixed problem with enum used as factory.


Bugs: ATLAS-1825
    https://issues.apache.org/jira/browse/ATLAS-1825


Repository: atlas


Description
-------

Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios
and solution approach.

Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of
entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string
of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
    * _replace_ Implements find and replace on attribute values.
    * _lowercase_ Converts attribute value to lower case. 
* Additional classes that unit test these classes have been added.

Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity
that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters
as well as the binary input from the file.
* Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.

CURL
====

*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{
    "options": {},
    "importTransform": {
        "transforms": {
            "hive_table": {
                "qualifiedName": [
                    "lowercase",
                    "replace:@cl1:@cl2"
                ]
            }
        }
    }
}
```

```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache"
-F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```

Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.

Notice the output from the API, it has the _AtlasRequest_ that was passed as input.

Documentation
=============
Need to update TWIKI documents.


Diffs (updated)
-----

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION

  intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION

  intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION

  pom.xml a93b8ad9 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00

  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb 
  repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81

  repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java
PRE-CREATION 
  repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838

  webapp/pom.xml 4132912d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860 


Diff: https://reviews.apache.org/r/56959/diff/6/

Changes: https://reviews.apache.org/r/56959/diff/5-6/


Testing
-------

**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.

**Functional Test**
* Small size data export & import.
* Medium size data export & import.

**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.


Thanks,

Ashutosh Mestry


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message