airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koen Mevissen <kmevis...@travix.com>
Subject Re: Data lineage and data portal
Date Wed, 29 Nov 2017 06:09:13 GMT
+1

I'm interested as well!



Op di 28 nov. 2017 om 14:04 schreef Marc Bollinger <marc@lumoslabs.com>

> +1
>
> On Mon, Nov 27, 2017 at 6:18 PM, Ruslan Dautkhanov <dautkhanov@gmail.com>
> wrote:
>
> > ‘’’
> > I'm
> > now working on sql scanners, extractors and other tools that allow me to
> > populate the database
> > ‘’’
> >
> > Very cool. Cloudera Navigator ( not an open source product) does this too
> > to some extent - collect metadata and create data lineage automatically (
> > stored as a Solr collection) by parsing sql queries.
> >
> > https://www.cloudera.com/documentation/enterprise/5-12-
> > x/topics/datamgmt_extraction_indexing.html
> >
> >
> >
> > On Mon, Nov 27, 2017 at 12:38 PM Gerard Toonstra <gtoonstra@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > So something that really drew my attention recently is a "data portal"
> > as
> > > described by a team from airbnb somewhere in May. The idea is
> basically a
> > > "facebook of data":
> > >
> > >
> > >
> > > https://medium.com/airbnb-engineering/democratizing-
> > data-at-airbnb-852d76c51770
> > >
> > >
> > > Unfortunately it looks like it's not going to be opensourced due to how
> > > heavily integrated it is with their specific infrastructure; but the
> idea
> > > itself to me sounds like it's something every organization of a certain
> > > size should have to keep track of data and stay informed as an
> > > organization.
> > >
> > > Based on the descriptions, I prototyped some things away and am happy
> > with
> > > the results and the speed that something like this can be constructed.
> > I'm
> > > now working on sql scanners, extractors and other tools that allow me
> to
> > > populate the database and put a poc together on some real data.
> > >
> > > If other people have similar concerns in their organization and think
> > this
> > > would be a great thing to have, reply to me or the list; with
> sufficient
> > > interest I may set up a web chat/meet session so this can be discussed
> in
> > > more detail and find ways to progress this.
> > >
> > >
> > > Best regards,
> > >
> > > Gerard
> > >
> >
>
-- 
Kind regards,
Met vriendelijke groet,

*Koen Mevissen*
Principal BI Developer


*Travix Nederland B.V.*
Piet Heinkade 55
1019 GM Amsterdam
The Netherlands

T. +31 (0)20 203 3241
E: KMevissen@travix.com
www.travix.com

*Brands: * CheapTickets  |  Vliegwinkel  |  Vayama  |  BudgetAir  |
 Flugladen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message