nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (NIFI-3709) Export NiFi flow dataset lineage to Apache Atlas
Date Tue, 05 Sep 2017 21:05:00 GMT


ASF GitHub Bot commented on NIFI-3709:

Github user gjlawran commented on the issue:
    We are also looking for assistance with bridging metadata from NiFi into Apache Atlas
- and we have posted a [paid opportunity ]( those
with an interest.  

> Export NiFi flow dataset lineage to Apache Atlas
> ------------------------------------------------
>                 Key: NIFI-3709
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Koji Kawamura
>            Assignee: Koji Kawamura
> While Apache NiFi has provenance and event level lineage support within its data flow,
Apache Atlas also does manage lineage between dataset and process those interacting with such
> It would be beneficial for users who use both NiFi and Atlas and if they can see end-to-end
data lineage on Atlas lineage graph, as some type of dataset are processed by both NiFi and
technologies around Atlas such as Storm, Falcon or Sqoop. For example, Kafka topics and Hive
> In order to make this integration happen, I propose a NiFi reporting task that analyzes
NiFi flow then creates DataSet and Process entities in Atlas.
> The challenge is how to design NiFi flow dataset level lineage within Atlas lineage graph.
> If we just add a single NiFi process and connect every DataSet from/to it, it would be
too ambiguous since it won't be clear which part of a NiFi flow actually interact with certain
> But if we put every NiFi processor as independent process in Atlas, it would be too granular,
too. Also, we already have detailed event level lineage in NiFi, we wouldn't need the same
level in Atlas.
> If we can group certain processors in a NiFI flow as a process in Atlas, it would be
a nice granularity.

This message was sent by Atlassian JIRA

View raw message