airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Airflow DAG Serialisation
Date Fri, 26 Jul 2019 15:54:12 GMT
Great! That's definitely one of the most painful aspects of Airflow. Happy
to help/comment/take part in the discussions and later in the
implementation.

On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong <xd.deng.r@gmail.com> wrote:

> Yes, it’s something really worth exploring.
>
> Recently I just finished a POC on DAG serialisation as well, while the
> main objective was to allow “rewriting” some components, like worker (for
> specific operators/sensors), in languages other than Python. We’re also
> choosing JSON, so that most languages can easily process the information
> from DAGs.
>
> Happy to know how this topic will be processed.
>
>
> XD
>
> > On 26 Jul 2019, at 10:36 PM, Kaxil Naik <kaxilnaik@gmail.com> wrote:
> >
> > Hi all,
> >
> > We, at Astronomer, are going to spend time working on DAG Serialisation.
> > There are 2 AIPs that are somewhat related to what we plan to work on:
> >
> >   - AIP-18 Persist all information from DAG file in DB
> >   <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB
> >
> >   - AIP-19 Making the webserver stateless
> >   <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless
> >
> >
> > We plan to use JSON as the Serialisation format and store it as a blob in
> > metadata DB.
> >
> > *Goals:*
> >
> >   - Make Webserver Stateless
> >   - Use the same version of the DAG across Webserver & Scheduler
> >   - Keep backward compatibility and have a flag (globally & at DAG level)
> >   to turn this feature on/off
> >   - Enable DAG Versioning (extended Goal)
> >
> >
> > We will be preparing a proposal (AIP) after some research and some
> initial
> > work and open it for the suggestions of the community.
> >
> > We already had some good brain-storming sessions with Twitter folks
> (DanD &
> > Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which
> will
> > be a good starting point for us.
> >
> > If anyone in the community is interested in it or has some experience
> about
> > the same and want to collaborate please let me know and join
> > #dag-serialisation channel on Airflow Slack.
> >
> > Regards,
> > Kaxil
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message