airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: UI to author flows
Date Sun, 09 Jul 2017 05:01:41 GMT
One option is to save the metadata in a relational database from your UI
and read from there as you create the DAG. The challenge might be around
change management where you may want to recreate things that you get for
free in source control.

To get the best of both worlds (UI and DSL in source control) I've seen it
done where a UI generates Python code and submits a PR on your behalf. That
allows for the user of the UI to learn the underlying DSL and follows the
same process (code review) as other workflow authors.

Max

On Fri, Jul 7, 2017 at 9:05 AM, Arpan Dhandhania <arpan@infoworks.io> wrote:

> Hi,
>
> We are building a web-based UI to author flows. A user will drag-n-drop
> tasks on a canvas and link them to define the dependency. In Airflow, these
> tasks are PythonOperators that invoke functions containing the business
> logic (written by us).
>
> I was wondering what is the best way to submit the UI-generated DAG to
> Airflow. As a test, we generated Python code and saved it to a file in the
> DAGs directory. While that works, I don’t think it is the best solution.
>
> Is there an API to define the DAG? Could Jinja templating help?
>
> Any recommendations of how to do this elegantly will be appreciated.
>
> Arpan Dhandhania
> e: arpan@infoworks.io
> m: +91 782 939 5618
>
> —
> Perfection is achieved, not when there is nothing more to add, but when
> there is nothing left to take away.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message