atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shwetha GS <sshivalingamur...@hortonworks.com>
Subject Re: Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export
Date Wed, 06 Apr 2016 11:21:46 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/#review127310
-----------------------------------------------------------




addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java (line
472)
<https://reviews.apache.org/r/45784/#comment190571>

    We need to fix the clusterName mess later - can't pickup hdfs clustername from hive conf



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 220)
<https://reviews.apache.org/r/45784/#comment190572>

    Earlier one was more readable. You can use set methods instead of this long constructor?



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 454)
<https://reviews.apache.org/r/45784/#comment190574>

    Aren't there cases where input/output is local fs, for example load from local path?



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 493)
<https://reviews.apache.org/r/45784/#comment190573>

    This should be part of HiveMetaStoreBridge and should be used in import-hive as well?
    
    Because this lineage will be created in import-hive, process name should be just tablename
for create table so that its created just once.



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java (line 165)
<https://reviews.apache.org/r/45784/#comment190575>

    Use these in Process type definition. 
    
    Actually, these should be in AtlasClient?


- Shwetha GS


On April 5, 2016, 11:58 p.m., Suma Shivaprasad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45784/
> -----------------------------------------------------------
> 
> (Updated April 5, 2016, 11:58 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-527
>     https://issues.apache.org/jira/browse/ATLAS-527
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Added support to track lineage between HDFS Paths and hive tables  in 
> 
> a. LOAD( at table, partition level) - input is a HDFS path and output is table( even
though we dont create partition entities, we are still tracking the lineage at table level
for partitions. This could be an issue if there are large number of partition queries which
is not being addressed in this jira - https://issues.apache.org/jira/browse/ATLAS-619) . refer
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
> b. IMPORT, EXPORT to and from hdfs paths - Refer https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
> c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
> d. ALTER TABLE LOCATION for an external table - input is the new hdfs path and o/p is
the table.
> 
> Also changed the ordering of model registration by sorting them by modifiedTime to ensure
they are registered in correct order
> 
> 
> Diffs
> -----
> 
>   addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
555d565 
>   addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala c964f73

>   addons/hive-bridge/pom.xml e125f18 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
3a802d7 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 68e32ff 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java e17afb8

>   addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 5665856

>   repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 0a04c5f

>   repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 430bb6b

> 
> Diff: https://reviews.apache.org/r/45784/diff/
> 
> 
> Testing
> -------
> 
> Added tests in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message