atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suma Shivaprasad <sumasai.shivapra...@gmail.com>
Subject Re: Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export
Date Wed, 06 Apr 2016 18:10:01 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/
-----------------------------------------------------------

(Updated April 6, 2016, 6:10 p.m.)


Review request for atlas.


Bugs: ATLAS-527
    https://issues.apache.org/jira/browse/ATLAS-527


Repository: atlas


Description
-------

Added support to track lineage between HDFS Paths and hive tables  in 

a. LOAD( at table, partition level) - input is a HDFS path and output is table( even though
we dont create partition entities, we are still tracking the lineage at table level for partitions.
This could be an issue if there are large number of partition queries which is not being addressed
in this jira - https://issues.apache.org/jira/browse/ATLAS-619) . refer https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
b. IMPORT, EXPORT to and from hdfs paths - Refer https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
d. ALTER TABLE LOCATION for an external table - input is the new hdfs path and o/p is the
table.

Also changed the ordering of model registration by sorting them by modifiedTime to ensure
they are registered in correct order


Diffs (updated)
-----

  addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java 555d565

  addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala c964f73 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 3a802d7

  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 68e32ff 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java e17afb8 
  addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 5665856

  client/src/main/java/org/apache/atlas/AtlasClient.java c3b4ba9 
  repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 430bb6b 

Diff: https://reviews.apache.org/r/45784/diff/


Testing
-------

Added tests in HiveHookIT


Thanks,

Suma Shivaprasad


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message