airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arpan Dhandhania <ar...@infoworks.io>
Subject Re: UI to author flows
Date Mon, 10 Jul 2017 08:26:57 GMT
Thanks Max. We are currently storing the metadata in our DB and reading off that. 

Since this setup will be exclusively used via our UI, we don't have a VCS configured, but
I agree with the idea of generating the DSL so that the user can learn how Airflow works.

I was hoping there would be an option other than generating Python code, but looks like there
isn't any.

I saw a section in the documentation titled experimental API. Is there a possibility that
an API will be exposed to author workflows?

Sent from my iPhone

> On Jul 9, 2017, at 10:31 AM, Maxime Beauchemin <maximebeauchemin@gmail.com> wrote:
> 
> One option is to save the metadata in a relational database from your UI
> and read from there as you create the DAG. The challenge might be around
> change management where you may want to recreate things that you get for
> free in source control.
> 
> To get the best of both worlds (UI and DSL in source control) I've seen it
> done where a UI generates Python code and submits a PR on your behalf. That
> allows for the user of the UI to learn the underlying DSL and follows the
> same process (code review) as other workflow authors.
> 
> Max
> 
>> On Fri, Jul 7, 2017 at 9:05 AM, Arpan Dhandhania <arpan@infoworks.io> wrote:
>> 
>> Hi,
>> 
>> We are building a web-based UI to author flows. A user will drag-n-drop
>> tasks on a canvas and link them to define the dependency. In Airflow, these
>> tasks are PythonOperators that invoke functions containing the business
>> logic (written by us).
>> 
>> I was wondering what is the best way to submit the UI-generated DAG to
>> Airflow. As a test, we generated Python code and saved it to a file in the
>> DAGs directory. While that works, I don’t think it is the best solution.
>> 
>> Is there an API to define the DAG? Could Jinja templating help?
>> 
>> Any recommendations of how to do this elegantly will be appreciated.
>> 
>> Arpan Dhandhania
>> e: arpan@infoworks.io
>> m: +91 782 939 5618
>> 
>> —
>> Perfection is achieved, not when there is nothing more to add, but when
>> there is nothing left to take away.
>> 
>> 

Mime
View raw message