gobblin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Tiwari (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (GOBBLIN-587) Implement partition level lineage for fs based destination
Date Fri, 30 Nov 2018 06:35:00 GMT

     [ https://issues.apache.org/jira/browse/GOBBLIN-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Abhishek Tiwari updated GOBBLIN-587:
------------------------------------
    Fix Version/s: 0.14.0

> Implement partition level lineage for fs based destination
> ----------------------------------------------------------
>
>                 Key: GOBBLIN-587
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-587
>             Project: Apache Gobblin
>          Issue Type: Task
>            Reporter: Zhixiong Chen
>            Assignee: Zhixiong Chen
>            Priority: Major
>             Fix For: 0.14.0
>
>
> Currently, gobblin lineage is sent at dataset level. The task is to send partition level
lineage for fs sink. An example kafka-hdfs partition lineage is
> {code:java}
> {
>   "timestamp": 1536785248451,
>   "namespace": {
>     "string": "gobblin.event.lineage"
>   },
>   "name": "LoginEvent",
>   "metadata": {
>     "destination": "{\"object-type\":\"org.apache.gobblin.dataset.PartitionDescriptor\",\"object-data\":{\"dataset\":{\"object-type\":\"org.apache.gobblin.dataset.DatasetDescriptor\",\"object-data\":{\"platform\":\"hdfs\",\"metadata\":{\"branch\":\"0\"},\"name\":\"/data/tracking/LoginEvent\"}},\"name\":\"hourly/2018/09/12/12\"}}",
>     "eventType": "LineageEvent",
>     "source": "{\"object-type\":\"org.apache.gobblin.dataset.DatasetDescriptor\",\"object-data\":{\"platform\":\"kafka\",\"metadata\":{},\"name\":\"LoginEvent\"}}",
>     "metricContextName": "org.apache.gobblin.runtime.SafeDatasetCommit.1693032310",
>     "metricContextID": "1a7895b0-9e93-414e-ac0b-038f9375c82e",
>     "class": "org.apache.gobblin.runtime.SafeDatasetCommit",
>   }
> }
> {code}
> {color:#d04437}*Note*{color}: Lineage is not available automatically. You might have
to implement the support in your source-destination pair.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message