airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Toonstra <gtoons...@gmail.com>
Subject Data lineage and data portal
Date Mon, 27 Nov 2017 19:38:23 GMT
Hi all,

So something that really drew my attention recently is a "data portal"  as
described by a team from airbnb somewhere in May. The idea is basically a
"facebook of data":


https://medium.com/airbnb-engineering/democratizing-data-at-airbnb-852d76c51770


Unfortunately it looks like it's not going to be opensourced due to how
heavily integrated it is with their specific infrastructure; but the idea
itself to me sounds like it's something every organization of a certain
size should have to keep track of data and stay informed as an organization.

Based on the descriptions, I prototyped some things away and am happy with
the results and the speed that something like this can be constructed. I'm
now working on sql scanners, extractors and other tools that allow me to
populate the database and put a poc together on some real data.

If other people have similar concerns in their organization and think this
would be a great thing to have, reply to me or the list; with sufficient
interest I may set up a web chat/meet session so this can be discussed in
more detail and find ways to progress this.


Best regards,

Gerard

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message