Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@atlas.incubator.apache.org
Content-Type: multipart/alternative;
 boundary="===============5423082365507916644=="
MIME-Version: 1.0
Subject: Re: Review Request 56959: Import API: Added Support for Transforming
 Entities During Import
From: Ashutosh Mestry <amestry@hortonworks.com>
To: Madhan Neethiraj <madhan@apache.org>
Cc: Ashutosh Mestry <amestry@hortonworks.com>,
 atlas <dev@atlas.incubator.apache.org>, Apoorv Naik <naik.apoorv@gmail.com>
Date: Thu, 25 May 2017 05:16:22 -0000
Message-ID: <20170525051622.61833.35088@reviews-vm2.apache.org>
Auto-Submitted: auto-generated
Sender: Ashutosh Mestry <noreply@reviews.apache.org>
References: <20170524231551.1142.63235@reviews-vm2.apache.org>
In-Reply-To: <20170524231551.1142.63235@reviews-vm2.apache.org>
Reply-To: Ashutosh Mestry <amestry@hortonworks.com>
archived-at: Thu, 25 May 2017 05:16:28 -0000

--===============5423082365507916644==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------

(Updated May 25, 2017, 5:16 a.m.)


Review request for atlas and Madhan Neethiraj.


Changes
-------

Fixed problem with enum used as factory.


Bugs: ATLAS-1825
    https://issues.apache.org/jira/browse/ATLAS-1825


Repository: atlas


Description
-------

Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach.

Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. They are:
    * _replace_ Implements find and replace on attribute values.
    * _lowercase_ Converts attribute value to lower case. 
* Additional classes that unit test these classes have been added.

Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file.
* Please take note of _ImportTransformerEnum_. This is first case where Enum has state variables.

CURL
====

*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{
    "options": {},
    "importTransform": {
        "transforms": {
            "hive_table": {
                "qualifiedName": [
                    "lowercase",
                    "replace:@cl1:@cl2"
                ]
            }
        }
    }
}
```

```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"
```

Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore.

Notice the output from the API, it has the _AtlasRequest_ that was passed as input.

Documentation
=============
Need to update TWIKI documents.


Diffs (updated)
-----

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java PRE-CREATION 
  intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java PRE-CREATION 
  pom.xml a93b8ad9 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb 
  repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81 
  repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java d4900838 
  webapp/pom.xml 4132912d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860 


Diff: https://reviews.apache.org/r/56959/diff/6/

Changes: https://reviews.apache.org/r/56959/diff/5-6/


Testing
-------

**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.

**Functional Test**
* Small size data export & import.
* Medium size data export & import.

**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.


Thanks,

Ashutosh Mestry


--===============5423082365507916644==--