atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vimal Sharma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-642) import-hive should create the lineage for external tables
Date Thu, 09 Jun 2016 05:58:21 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321952#comment-15321952
] 

Vimal Sharma commented on ATLAS-642:
------------------------------------

[~suma.shivaprasad]
It makes sense to use the same function getProcessQualifiedName() in HiveHook to get the process
qualified name. However we need "SHOW CREATE TABLE" query to get the same qualified name as
in HiveHook. The reason is that if qualified name differs then import_hive.sh would end up
creating a new process and we will have two different lineage graphs for the same table.

Process Referenceable creation in HiveHook uses a HiveEvent to get the process details. We
don't have a HiveEvent in HMSB.

> import-hive should create the lineage for external tables
> ---------------------------------------------------------
>
>                 Key: ATLAS-642
>                 URL: https://issues.apache.org/jira/browse/ATLAS-642
>             Project: Atlas
>          Issue Type: Sub-task
>            Reporter: Suma Shivaprasad
>            Assignee: Vimal Sharma
>             Fix For: 0.7-incubating
>
>         Attachments: ATLAS-642.patch
>
>
> import-hive can also start showing lineage between HDFS paths and tables. 
> However the create table process should also register the exact query that was  used
to create the table. Show create table does that by reconstructing the query. Need to figure
out if we can potentially use that to create the query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message