atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From afly...@163.com
Subject Review Request 68883: ATLAS-2893?Atlas Column Lineage of Hive Hook to support the hive old version
Date Sat, 29 Sep 2018 06:56:03 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68883/
-----------------------------------------------------------

Review request for atlas.


Repository: atlas


Description
-------

Right now atlas hook for apache hive of column lineage is limited, only "Column level lineage
works with Hive version 1.2.1 after the patch for HIVE-13112 is applied to Hive source" could
support the hive column lineage. In some product env, hive version still very old, but we
should also support the hive column lineage.


Diffs
-----

  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
ae01d504d 


Diff: https://reviews.apache.org/r/68883/diff/1/


Testing
-------

I refer to the hive source code, when we set the altas hook class at the hive-site.xml with
the name 'org.apache.atlas.hive.hook.HiveHook', hive optimizer would not set the linegae info
into the hookContext, after the patch HIVE-13112 already fix this. But early hive version
still could not work, so i go ahead and dive into the hive lineageState of SessionState, maybe
we could get the private info of the LineageCtx to work out the lineage dependency,  then
get the column lineage.
   
   Under the Atlas hive-bridge module, i modified the CreateHiveProcess Class, and added the
method to get the column lineage from the hive lineageState of SessionState, it could work
with the pre-released hive version before hive-1.2.1. I test the code with the hive version
1.2.0 and 0.12.0, it could get the hive column lineage correctly.
   
   The hive sql which i test are below:

   CREATE TABLE table1(id int, name string, age int, address string) ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',' STORED AS TEXTFILE;
   CREATE TABLE table2 AS SELECT id,name,age,address FROM table1;


Thanks,

aflyary


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message