airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radek Tomšej <ra...@tomsej.cz>
Subject Re: Data lineage and data portal
Date Tue, 28 Nov 2017 21:15:41 GMT
Hi I am interested too. I have tried to make a POC with Elasticsearch +
Kibana so would be nice to share some experience.


On 2017-11-28 00:45, Gurer Kiratli wrote:
> If there are particular questions about the Data Portal, I would be
happy>
> to get a list of these and work on looping in the Data Portal folks from>
> Airbnb.>
>
> Cheers,>
> Gurer>
>
> On Mon, Nov 27, 2017 at 2:41 PM, Megan Kearl wrote:>
>
> > I'm interested too>
> >>
> > On Nov 27, 2017 3:26 PM, "Bolke de Bruin" wrote:>
> >>
> > > Natuurlijk :-)>
> > >>
> > > Absolutely!>
> > >>
> > > Sent from my iPhone>
> > >>
> > > > On 27 Nov 2017, at 21:23, Chris Riccomini >
> > wrote:>
> > > >>
> > > > Interested>
> > > >>
> > > >> On Mon, Nov 27, 2017 at 12:07 PM, Kerr Shireman >
> > > wrote:>
> > > >>>
> > > >> I am interested. I remember being pretty excited when I read that>
> > blog>
> > > >> post.>
> > > >> On Mon, Nov 27, 2017 at 2:00 PM Arthur Wiedmer <>
> > > arthur.wiedmer@gmail.com>>
> > > >> wrote:>
> > > >>>
> > > >>> Likewise!>
> > > >>>>
> > > >>> Best,>
> > > >>> Arthur>
> > > >>>>
> > > >>> On Mon, Nov 27, 2017 at 11:57 AM, Alison Stanton <>
> > > >>> astanton@bankofknowledge.net> wrote:>
> > > >>>>
> > > >>>> I'd like to be kept informed.>
> > > >>>>>
> > > >>>> Alison Stanton>
> > > >>>>>
> > > >>>> On Mon, Nov 27, 2017 at 1:53 PM, Laura Lorenz <>
> > > >> llorenz@industrydive.com>>
> > > >>>> wrote:>
> > > >>>>>
> > > >>>>> We're definitely looking for something like that here,
so I
would>
> > > >> like>
> > > >>> to>
> > > >>>>> jump in on this discussion.>
> > > >>>>>>
> > > >>>>> Laura>
> > > >>>>>>
> > > >>>>> On Mon, Nov 27, 2017 at 2:38 PM, Gerard Toonstra <>
> > > >> gtoonstra@gmail.com>>
> > > >>>>> wrote:>
> > > >>>>>>
> > > >>>>>> Hi all,>
> > > >>>>>>>
> > > >>>>>> So something that really drew my attention recently
is a
"data>
> > > >>> portal">
> > > >>>>> as>
> > > >>>>>> described by a team from airbnb somewhere in May.
The idea is>
> > > >>>> basically a>
> > > >>>>>> "facebook of data":>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>> https://medium.com/airbnb-engineering/democratizing->
> > > >>>>>> data-at-airbnb-852d76c51770>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>> Unfortunately it looks like it's not going to be opensourced
due>
> > to>
> > > >>> how>
> > > >>>>>> heavily integrated it is with their specific infrastructure;
but>
> > > >> the>
> > > >>>> idea>
> > > >>>>>> itself to me sounds like it's something every organization
of
a>
> > > >>> certain>
> > > >>>>>> size should have to keep track of data and stay informed
as
an>
> > > >>>>>> organization.>
> > > >>>>>>>
> > > >>>>>> Based on the descriptions, I prototyped some things
away and
am>
> > > >> happy>
> > > >>>>> with>
> > > >>>>>> the results and the speed that something like this
can be>
> > > >>> constructed.>
> > > >>>>> I'm>
> > > >>>>>> now working on sql scanners, extractors and other
tools that
allow>
> > > >> me>
> > > >>>> to>
> > > >>>>>> populate the database and put a poc together on some
real
data.>
> > > >>>>>>>
> > > >>>>>> If other people have similar concerns in their organization
and>
> > > >> think>
> > > >>>>> this>
> > > >>>>>> would be a great thing to have, reply to me or the
list; with>
> > > >>>> sufficient>
> > > >>>>>> interest I may set up a web chat/meet session so this
can be>
> > > >>> discussed>
> > > >>>> in>
> > > >>>>>> more detail and find ways to progress this.>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>> Best regards,>
> > > >>>>>>>
> > > >>>>>> Gerard>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message