airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Airflow DAG Serialisation
Date Fri, 26 Jul 2019 17:26:36 GMT
Great to see this happening!

On Fri, Jul 26, 2019 at 8:54 AM Jarek Potiuk <Jarek.Potiuk@polidea.com>
wrote:

> Great! That's definitely one of the most painful aspects of Airflow. Happy
> to help/comment/take part in the discussions and later in the
> implementation.
>
> On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong <xd.deng.r@gmail.com> wrote:
>
> > Yes, it’s something really worth exploring.
> >
> > Recently I just finished a POC on DAG serialisation as well, while the
> > main objective was to allow “rewriting” some components, like worker (for
> > specific operators/sensors), in languages other than Python. We’re also
> > choosing JSON, so that most languages can easily process the information
> > from DAGs.
> >
> > Happy to know how this topic will be processed.
> >
> >
> > XD
> >
> > > On 26 Jul 2019, at 10:36 PM, Kaxil Naik <kaxilnaik@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > We, at Astronomer, are going to spend time working on DAG
> Serialisation.
> > > There are 2 AIPs that are somewhat related to what we plan to work on:
> > >
> > >   - AIP-18 Persist all information from DAG file in DB
> > >   <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> > >
> > >   - AIP-19 Making the webserver stateless
> > >   <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> > >
> > >
> > > We plan to use JSON as the Serialisation format and store it as a blob
> in
> > > metadata DB.
> > >
> > > *Goals:*
> > >
> > >   - Make Webserver Stateless
> > >   - Use the same version of the DAG across Webserver & Scheduler
> > >   - Keep backward compatibility and have a flag (globally & at DAG
> level)
> > >   to turn this feature on/off
> > >   - Enable DAG Versioning (extended Goal)
> > >
> > >
> > > We will be preparing a proposal (AIP) after some research and some
> > initial
> > > work and open it for the suggestions of the community.
> > >
> > > We already had some good brain-storming sessions with Twitter folks
> > (DanD &
> > > Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which
> > will
> > > be a good starting point for us.
> > >
> > > If anyone in the community is interested in it or has some experience
> > about
> > > the same and want to collaborate please let me know and join
> > > #dag-serialisation channel on Airflow Slack.
> > >
> > > Regards,
> > > Kaxil
> >
> >
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message