From dev-return-2355-archive-asf-public=cust-asf.ponee.io@gobblin.incubator.apache.org Fri Nov 30 07:35:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 97858180671 for ; Fri, 30 Nov 2018 07:35:04 +0100 (CET) Received: (qmail 10836 invoked by uid 500); 30 Nov 2018 06:35:03 -0000 Mailing-List: contact dev-help@gobblin.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@gobblin.incubator.apache.org Delivered-To: mailing list dev@gobblin.incubator.apache.org Received: (qmail 10759 invoked by uid 99); 30 Nov 2018 06:35:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Nov 2018 06:35:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 35018C047E for ; Fri, 30 Nov 2018 06:35:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 5fEmWm9dTGDV for ; Fri, 30 Nov 2018 06:35:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id A81705FB32 for ; Fri, 30 Nov 2018 06:35:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 45FF5E0F4C for ; Fri, 30 Nov 2018 06:35:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 62D0327770 for ; Fri, 30 Nov 2018 06:35:00 +0000 (UTC) Date: Fri, 30 Nov 2018 06:35:00 +0000 (UTC) From: "Abhishek Tiwari (JIRA)" To: dev@gobblin.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (GOBBLIN-587) Implement partition level lineage for fs based destination MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/GOBBLIN-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Tiwari updated GOBBLIN-587: ------------------------------------ Fix Version/s: 0.14.0 > Implement partition level lineage for fs based destination > ---------------------------------------------------------- > > Key: GOBBLIN-587 > URL: https://issues.apache.org/jira/browse/GOBBLIN-587 > Project: Apache Gobblin > Issue Type: Task > Reporter: Zhixiong Chen > Assignee: Zhixiong Chen > Priority: Major > Fix For: 0.14.0 > > > Currently, gobblin lineage is sent at dataset level. The task is to send partition level lineage for fs sink. An example kafka-hdfs partition lineage is > {code:java} > { > "timestamp": 1536785248451, > "namespace": { > "string": "gobblin.event.lineage" > }, > "name": "LoginEvent", > "metadata": { > "destination": "{\"object-type\":\"org.apache.gobblin.dataset.PartitionDescriptor\",\"object-data\":{\"dataset\":{\"object-type\":\"org.apache.gobblin.dataset.DatasetDescriptor\",\"object-data\":{\"platform\":\"hdfs\",\"metadata\":{\"branch\":\"0\"},\"name\":\"/data/tracking/LoginEvent\"}},\"name\":\"hourly/2018/09/12/12\"}}", > "eventType": "LineageEvent", > "source": "{\"object-type\":\"org.apache.gobblin.dataset.DatasetDescriptor\",\"object-data\":{\"platform\":\"kafka\",\"metadata\":{},\"name\":\"LoginEvent\"}}", > "metricContextName": "org.apache.gobblin.runtime.SafeDatasetCommit.1693032310", > "metricContextID": "1a7895b0-9e93-414e-ac0b-038f9375c82e", > "class": "org.apache.gobblin.runtime.SafeDatasetCommit", > } > } > {code} > {color:#d04437}*Note*{color}: Lineage is not available automatically. You might have to implement the support in your source-destination pair. -- This message was sent by Atlassian JIRA (v7.6.3#76005)