Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8A1B9200B25 for ; Wed, 8 Jun 2016 14:43:25 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 88AF8160A2E; Wed, 8 Jun 2016 12:43:25 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D36D3160A29 for ; Wed, 8 Jun 2016 14:43:24 +0200 (CEST) Received: (qmail 48801 invoked by uid 500); 8 Jun 2016 12:43:23 -0000 Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.incubator.apache.org Delivered-To: mailing list dev@atlas.incubator.apache.org Received: (qmail 48790 invoked by uid 99); 8 Jun 2016 12:43:23 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2016 12:43:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 8298AC2843 for ; Wed, 8 Jun 2016 12:43:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -5.446 X-Spam-Level: X-Spam-Status: No, score=-5.446 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id FOiFeKRv36a8 for ; Wed, 8 Jun 2016 12:43:23 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id F3C3F5F23E for ; Wed, 8 Jun 2016 12:43:21 +0000 (UTC) Received: (qmail 48351 invoked by uid 99); 8 Jun 2016 12:43:21 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2016 12:43:21 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 1FE202C1F60 for ; Wed, 8 Jun 2016 12:43:21 +0000 (UTC) Date: Wed, 8 Jun 2016 12:43:21 +0000 (UTC) From: "Vimal Sharma (JIRA)" To: dev@atlas.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ATLAS-642) import-hive should create the lineage for external tables MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 08 Jun 2016 12:43:25 -0000 [ https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320469#comment-15320469 ] Vimal Sharma commented on ATLAS-642: ------------------------------------ We can create the lineage by extracting metadata from Hive table. We need to create a hive process with HDFS path and table as input and output respectively. I have submit a patch with this change. We can revisit the task of getting the "SHOW CREATE TABLE" query for a table in another JIRA. > import-hive should create the lineage for external tables > --------------------------------------------------------- > > Key: ATLAS-642 > URL: https://issues.apache.org/jira/browse/ATLAS-642 > Project: Atlas > Issue Type: Sub-task > Reporter: Suma Shivaprasad > Assignee: Vimal Sharma > Fix For: 0.7-incubating > > Attachments: ATLAS-642.patch > > > import-hive can also start showing lineage between HDFS paths and tables. > However the create table process should also register the exact query that was used to create the table. Show create table does that by reconstructing the query. Need to figure out if we can potentially use that to create the query -- This message was sent by Atlassian JIRA (v6.3.4#6332)