Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@atlas.incubator.apache.org
Date: Thu, 7 Apr 2016 20:21:25 +0000 (UTC)
From: "Suma Shivaprasad (JIRA)" <jira@apache.org>
To: dev@atlas.incubator.apache.org
Message-ID: <JIRA.12944623.1456428926000.167490.1460060485520@Atlassian.JIRA>
In-Reply-To: <JIRA.12944623.1456428926000@Atlassian.JIRA>
References: <JIRA.12944623.1456428926000@Atlassian.JIRA>
 <JIRA.12944623.1456428926381@arcas>
Subject: [jira] [Commented] (ATLAS-528) support drop table, view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230983#comment-15230983 ] 

Suma Shivaprasad commented on ATLAS-528:
----------------------------------------

@Hemanth yamijala Thanks for the review.

The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important.

--> Raised ATLAS-653 to track this.

On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this?

--> HiveHookIT.validateProcess already check this by querying with the same exact query

If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later?

--> Raised ATLAS-652 to track this.

We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called?

--> I observed that the hooks is being called  and the hook is ignoring this since there are no outputs for this query. Added a testcase HiveHookIT.testDropNonExistingTable to test this. 

There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO.

--> Added 

> support drop table, view
> ------------------------
>
>                 Key: ATLAS-528
>                 URL: https://issues.apache.org/jira/browse/ATLAS-528
>             Project: Atlas
>          Issue Type: Sub-task
>    Affects Versions: 0.7-incubating
>            Reporter: Suma Shivaprasad
>            Assignee: Suma Shivaprasad
>             Fix For: 0.7-incubating
>
>         Attachments: ATLAS-528.1.patch, ATLAS-528.patch
>
>
> Drop table and view requires soft deletes. The reason is whenever a table is dropped , it also may have an associated lineage which consists of a hive_process which N input tables and an output table. If the table is dropped, the lineage edge is also dropped resulting in incorrect lineage history. 
> With soft deletes, the expected behaviour is to changes the table status to "deleted" and when the table is recreated through a create table statement, then create another vertex/entity for that table with the new state. Also,  the lineage for this newly recreated table should be a new hive_process and should not reuse the existing entity/vertex even though the hive statement for that process is the same.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)