airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Bollinger <m...@lumoslabs.com>
Subject Re: Data lineage and data portal
Date Tue, 28 Nov 2017 13:03:27 GMT
+1

On Mon, Nov 27, 2017 at 6:18 PM, Ruslan Dautkhanov <dautkhanov@gmail.com>
wrote:

> ‘’’
> I'm
> now working on sql scanners, extractors and other tools that allow me to
> populate the database
> ‘’’
>
> Very cool. Cloudera Navigator ( not an open source product) does this too
> to some extent - collect metadata and create data lineage automatically (
> stored as a Solr collection) by parsing sql queries.
>
> https://www.cloudera.com/documentation/enterprise/5-12-
> x/topics/datamgmt_extraction_indexing.html
>
>
>
> On Mon, Nov 27, 2017 at 12:38 PM Gerard Toonstra <gtoonstra@gmail.com>
> wrote:
>
> > Hi all,
> >
> > So something that really drew my attention recently is a "data portal"
> as
> > described by a team from airbnb somewhere in May. The idea is basically a
> > "facebook of data":
> >
> >
> >
> > https://medium.com/airbnb-engineering/democratizing-
> data-at-airbnb-852d76c51770
> >
> >
> > Unfortunately it looks like it's not going to be opensourced due to how
> > heavily integrated it is with their specific infrastructure; but the idea
> > itself to me sounds like it's something every organization of a certain
> > size should have to keep track of data and stay informed as an
> > organization.
> >
> > Based on the descriptions, I prototyped some things away and am happy
> with
> > the results and the speed that something like this can be constructed.
> I'm
> > now working on sql scanners, extractors and other tools that allow me to
> > populate the database and put a poc together on some real data.
> >
> > If other people have similar concerns in their organization and think
> this
> > would be a great thing to have, reply to me or the list; with sufficient
> > interest I may set up a web chat/meet session so this can be discussed in
> > more detail and find ways to progress this.
> >
> >
> > Best regards,
> >
> > Gerard
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message