airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marwan Nabil <mrwanbaghda...@gmail.com>
Subject Re: [Discuss] Airflow Kubernetes worker configuration should be parsed from YAML
Date Wed, 06 Mar 2019 16:49:33 GMT
Thanks for starting the discussion.

I think it would be great. And a great first step would be to supply the
filepath.

A decorator approach would be suitable I think and would allow for
extendability and would allow for much than yaml. I'm thinking helm charts
:D

On 2019/03/06 14:58:03, da...@ssense.com <d...@ssense.com> wrote:
> Hi,>
>
> I would like to discuss parsing YAML for the Kubernetes worker
configuration instead of the current process of programmatically generating
the YAML from the Pod and PodRequest Factory as is done currently.>
>
> *Motivation:*>
>
> Kubernetes configuration is quite complex. Instead of using the
configuration system that is offered natively by Kubernetes (YAML),the
current method involves programmatically recreating this YAML file. Fully
re-implementing the configuration in Airflow is taking a lot of time, and
at the moment many features available through YAML configuration are not
available in Airflow. Furthermore, as the Kubernetes API evolves, the
Airflow codebase will have to change with it, and Airflow will be in a
constant state of catching up with missing features available. This can all
be solved by simply parsing the YAML file.>
>
> *Idea:*>
>
> Either pass in the YAML as string or have a path to the YAML file.>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message