Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@atlas.incubator.apache.org
Content-Type: multipart/alternative;
 boundary="===============6556310753572658429=="
MIME-Version: 1.0
Subject: Re: Review Request 45784: Hve Hook - Support tracking lineage for
 External Tables( Create/alter) , Load, import, export
From: Suma Shivaprasad <sumasai.shivaprasad@gmail.com>
To: atlas <dev@atlas.incubator.apache.org>,
 Shwetha GS <sshivalingamurthy@hortonworks.com>
Date: Wed, 06 Apr 2016 17:44:27 -0000
Message-ID: <20160406174427.20106.82884@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: Suma Shivaprasad <noreply@reviews.apache.org>
References: <20160406170323.20106.50012@reviews.apache.org>
In-Reply-To: <20160406170323.20106.50012@reviews.apache.org>
Reply-To: Suma Shivaprasad <sumasai.shivaprasad@gmail.com>

--===============6556310753572658429==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/
-----------------------------------------------------------

(Updated April 6, 2016, 5:44 p.m.)


Review request for atlas.


Changes
-------

Fixed review comments


Bugs: ATLAS-527
    https://issues.apache.org/jira/browse/ATLAS-527


Repository: atlas


Description
-------

Added support to track lineage between HDFS Paths and hive tables  in 

a. LOAD( at table, partition level) - input is a HDFS path and output is table( even though we dont create partition entities, we are still tracking the lineage at table level for partitions. This could be an issue if there are large number of partition queries which is not being addressed in this jira - https://issues.apache.org/jira/browse/ATLAS-619) . refer https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
b. IMPORT, EXPORT to and from hdfs paths - Refer https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
d. ALTER TABLE LOCATION for an external table - input is the new hdfs path and o/p is the table.

Also changed the ordering of model registration by sorting them by modifiedTime to ensure they are registered in correct order


Diffs (updated)
-----

  addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java 555d565 
  addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala c964f73 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 3a802d7 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 68e32ff 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java e17afb8 
  addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 5665856 
  client/src/main/java/org/apache/atlas/AtlasClient.java c3b4ba9 
  repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 0a04c5f 
  repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 430bb6b 

Diff: https://reviews.apache.org/r/45784/diff/


Testing
-------

Added tests in HiveHookIT


Thanks,

Suma Shivaprasad


--===============6556310753572658429==--