atlas-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mad...@apache.org
Subject [06/14] incubator-atlas git commit: updated documentation for import API
Date Sun, 09 Jul 2017 15:07:44 GMT
updated documentation for import API

Signed-off-by: Madhan Neethiraj <madhan@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/incubator-atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-atlas/commit/958c48c2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-atlas/tree/958c48c2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-atlas/diff/958c48c2

Branch: refs/heads/feature-odf
Commit: 958c48c2c1807e68dd8cceda6857ec065c083a4b
Parents: 9a4ed46
Author: ashutoshm <amestry@hortonworks.com>
Authored: Fri Jun 30 16:54:12 2017 -0700
Committer: Madhan Neethiraj <madhan@apache.org>
Committed: Fri Jun 30 22:42:40 2017 -0700

----------------------------------------------------------------------
 docs/src/site/twiki/Export-API.twiki         |   2 +-
 docs/src/site/twiki/Export-HDFS-API.twiki    |   2 +-
 docs/src/site/twiki/Import-API-Options.twiki | 121 ++++++++++++++++++++++
 docs/src/site/twiki/Import-API.twiki         |  17 +--
 docs/src/site/twiki/Import-Export-API.twiki  |   7 +-
 5 files changed, 138 insertions(+), 11 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/958c48c2/docs/src/site/twiki/Export-API.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Export-API.twiki b/docs/src/site/twiki/Export-API.twiki
index 4027f3d..0df6629 100644
--- a/docs/src/site/twiki/Export-API.twiki
+++ b/docs/src/site/twiki/Export-API.twiki
@@ -138,7 +138,7 @@ The _metrics_ contains the number of types and entities exported as part
of the
 Below are sample CURL calls that demonstrate Export of _!QuickStart_ database.
 
 <verbatim>
-curl -X POST -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache"
-d '{
+curl -X POST -u adminuser:password -H "Content-Type: application/json" -H "Cache-Control:
no-cache" -d '{
     "itemsToExport": [
             { "typeName": "DB", "uniqueAttributes": { "name": "Sales" }
             { "typeName": "DB", "uniqueAttributes": { "name": "Reporting" }

http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/958c48c2/docs/src/site/twiki/Export-HDFS-API.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Export-HDFS-API.twiki b/docs/src/site/twiki/Export-HDFS-API.twiki
index 64ba123..6c20060 100644
--- a/docs/src/site/twiki/Export-HDFS-API.twiki
+++ b/docs/src/site/twiki/Export-HDFS-API.twiki
@@ -66,7 +66,7 @@ To export entities that represent HDFS path, use the Export API using the
_match
 Below are sample CURL calls that performs export operation on the _Sample HDFS Setup_ shown
above.
 
 <verbatim>
-curl -X POST -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache"
-d '{
+curl -X POST -u adminuser:password -H "Content-Type: application/json" -H "Cache-Control:
no-cache" -d '{
     "itemsToExport": [
             { "typeName": "hdfs_path", "uniqueAttributes": { "name": "FinanceAll" }
         }

http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/958c48c2/docs/src/site/twiki/Import-API-Options.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Import-API-Options.twiki b/docs/src/site/twiki/Import-API-Options.twiki
new file mode 100644
index 0000000..9acfee1
--- /dev/null
+++ b/docs/src/site/twiki/Import-API-Options.twiki
@@ -0,0 +1,121 @@
+---+ Import API Options
+
+Import API options are specified as _options_ JSON. Since the API accepts multi-part form
data, it is possible to sepecify multipls input streams within the CURL call.
+
+---+++ Examples Using CURL Calls
+<verbatim>
+curl -g -X POST -u adminuser:password -H "Content-Type: multipart/form-data"
+            -H "Cache-Control: no-cache"
+            -F request=@importOptions.json
+            -F data=@quickStartDB.zip
+            "http://localhost:21000/api/atlas/admin/import"
+</verbatim>
+
+To use the defaults, set the contents of _importOptions.json_ to:
+<verbatim>
+{
+  "options": {
+  }
+}
+</verbatim>
+
+
+---+++ Options
+Following options are supported for Import process:
+
+   * Specify transforms during import operation.
+   * Resume import by specifying starting entity guid.
+   * Optionally import type definition.
+
+---++++ Transforms
+
+During the import process, the attribute value of the incoming entity can be changed.
+
+This is possible by specifying entity type and at attribute to be modified and then the manner
in which it needs to be modified.
+
+Right now these are the transforms that can be applied:
+   * _lowercase_ Converts the attribute value to lower case.
+   * _replace_ This performs a string find and replace operation. It takes two parameters,
the first is the string to search for and the next one is the string to replace it with.
+
+Example:
+
+The example below applies couple of transforms to the the _qualifiedName_ attribute of hive_table.
It converts the value to lower case, then searches for 'cl1', if found, replaces it with 'cl2'.
+
+To use the option, set the contents of _importOptions.json_ to:
+<verbatim>
+{
+  "options": {
+
+   "transforms": {
+        "hive_table": {
+          "qualifiedName": [
+              "lowercase",
+              "replace:@cl1:@cl2"
+            ]
+          }
+        }
+  }
+}
+</verbatim>
+
+Please refer to [[https://issues.apache.org/jira/browse/ATLAS-1825][ATLAS-1825]] for details
scenarios when this option could be used.
+
+---++++ Start Guid or Start Index
+
+When an import operation is in progress and the server goes down, it would be possible to
resume import from the last successfully imported entity. This would allow the import to resume
from where it left off.
+
+Server-side logging is improved to display the detail of the last successfully imported entity,
this includes the index within the import list and the entity's guid. Either can be used specify
the point to resume import.
+
+To use the option, set the contents of _importOptions.json_ to:
+<verbatim>
+{
+
+  "options": {
+    "startGuid": "bd97c78e-3fa5-4f9c-9f48-3683ca3d1fb1"
+  }
+}
+</verbatim>
+
+To use _startIndex_, use the following in the _importOptions.json_:
+<verbatim>
+{
+
+  "options": {
+    "startIndex": "332"
+  }
+}
+</verbatim>
+
+Steps to use the behavior:
+   * Start an import (using the CURL) that is fairly long, say about 1000+ entities.
+   * While the import is in progress, stop atlas server (using atlas_stop.py).
+   * From the log file located at _/var/log/atlas/application.log_ get the last successfully
imported entity GUID or index position.
+   * Update the _importOptions.json_ with the guid.
+   * Restart import.
+
+---++++ Optional Importing Type Definition
+
+The output of Export has _atlas-typedef.json_ that contains the type definitions for the
entities exported.
+
+By default (that is if no options is specified), the type definitions are imported and applied
to the system. The entity import is then performed.
+
+In some cases, you would not want to modify the type definitions. Import may be better off
failing than the types be modified.
+
+This option allows for optionally importing of type definition. The option is set to _true_
by default, which means that type definition is imported. With this option set to _false_,
type definitions preseneraent in the source will not be imported. In case of mismatch between
the entities being imported the types present in the system where the import is being performed,
the operation will fail.
+
+Table below enumerates the conditions that get addressed as part of type definition import:
+
+|*Condition*|*Action*|
+| Incoming type does not exist in target system | Type is created. |
+|Type to be imported and type in target system are same | No change |
+|Type to be imported and type in target system differ by some attributes| Target system type
is updated to the attributes present in the source. It is possible that the target system
will have attributes in addition to the one present in the source. In that case, the target
system's type attributes will be an union of the attributes. (Attributes in target system
will not be deleted to match the source.)|
+
+To use the option, set the contents of _importOptions.json_ to:
+<verbatim>
+{
+
+  "options": {
+    "updateTypeDefinition": true
+  }
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/958c48c2/docs/src/site/twiki/Import-API.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Import-API.twiki b/docs/src/site/twiki/Import-API.twiki
index 7eedf19..b5de113 100644
--- a/docs/src/site/twiki/Import-API.twiki
+++ b/docs/src/site/twiki/Import-API.twiki
@@ -35,7 +35,7 @@ __Method Signature for Import__
 @POST
 @Path("/import")
 @Produces("application/json; charset=UTF-8")
-@Consumes("application/octet-stream")
+@Consumes("multipart/form-data")
 </verbatim>
 
 __Method Signature for Import File__
@@ -46,6 +46,9 @@ __Method Signature for Import File__
 @Consumes("application/json")
 </verbatim>
 
+__Import Options__
+Please see __[[Import-API-Options][here]]__ for the available options during import process.
+
 __!AtlasImportResult Response__
 The API will return the results of the import operation in the format defined by the _!AtlasImportResult_:
    * _!AtlasImportParameters_: This contains a collection of name value pair of the options
that are applied during the import operation.
@@ -56,14 +59,16 @@ The API will return the results of the import operation in the format
defined by
 ---+++ Examples Using CURL Calls
 The call below performs Import of _!QuickStart_ database using POST.
 <verbatim>
-curl -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H "Cache-Control:
no-cache"
-    --data-binary @quickStartDB.zip
-    "http://localhost:21000/api/atlas/admin/import" > quickStartDB-import-result.json
+curl -g -X POST -u adminuser:password -H "Content-Type: multipart/form-data"
+            -H "Cache-Control: no-cache"
+            -F request=@importOptions.json
+            -F data=@quickStartDB.zip
+            "http://localhost:21000/api/atlas/admin/import"
 </verbatim>
 
 The call below performs Import of _!QuickStart_ database using a ZIP file available on server.
 <verbatim>
-curl -X POST -u admin:admin -H "Cache-Control: no-cache"
+curl -X POST -u adminuser:password -H "Cache-Control: no-cache"
 "http://localhost:21000/api/atlas/admin/importFile?FILENAME=/root/quickStartDB.zip" >
quickStartDB-import-result.json
 </verbatim>
 
@@ -106,4 +111,4 @@ The _metrics_ contain a breakdown of the types and entities imported along
with
     ],
     "operationStatus": "SUCCESS"
 }
-</verbatim>
\ No newline at end of file
+</verbatim>

http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/958c48c2/docs/src/site/twiki/Import-Export-API.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Import-Export-API.twiki b/docs/src/site/twiki/Import-Export-API.twiki
index 519f208..6b07b64 100644
--- a/docs/src/site/twiki/Import-Export-API.twiki
+++ b/docs/src/site/twiki/Import-Export-API.twiki
@@ -28,6 +28,7 @@ If an import or export operation is initiated while another is in progress,
the
 Unhandled errors will be returned as Internal error code 500.
 
 ---++ REST API Reference
-   * __[[Export-API][Export API]]__
-   * __[[Export-HDFS-API][Export HDFS API]]__
-   * __[[Import-API][Import API]]__
+   * __[[Export-API][Export]]__
+   * __[[Export-HDFS-API][Export HDFS]]__
+   * __[[Import-API][Import]]__
+   * __[[Import-API-Options][Import Options]]__


Mime
View raw message