airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Contributing to the Kubernetes Exeuctor
Date Wed, 06 Feb 2019 19:40:31 GMT
Hi Darren

These improvements sound super useful and will benefit many. Starting to contribute is as
easy as opening a PR as you have done. There is no reason to ask for permission to the community.
Sometimes it might be handy to provide an explanation of what you try to accomplish particularly
if there are backward incompatible changes or paradigm changes. I don't see those popping
up in your list yet so it should be easy to get your much appreciated improvements in. 

Cheers
Bolke

Sent from my iPhone

> On 6 Feb 2019, at 10:41, Darren Haken <darrenhaken@gmail.com> wrote:
> 
> Hi,
> 
> My name is Darren and I'm head of data at Auto Trader UK. Auto Trader is
> the largest automotive marketplace, one of the largest sites in the UK
> (based on traffic).
> 
> We have been using Airflow for about 18 months with the LocalExecutor and
> now the K8 Executor is out we've been migrating towards that.
> 
> FYI we are heavy users of Kubernetes:
> 
>   - https://cloud.google.com/customers/auto-trader-uk/
>   -
>   https://cloudplatform.googleblog.com/2018/07/istio-reaches-1-0-ready-for-prod.html
> 
> Whilst working on the migration, we've found there are some improvements
> which would be good to make to the executor:
> 
>   - Support longer DAG names - we've opened a PR for this and are working
>   on it already. https://github.com/apache/airflow/pull/4636
>   - Improve the executor from a Kubernetes Operator perspective.
>      - Define requests and limits for Kubernetes for the workers. We
>      operate a multi-tenant cluster across teams and not being able to define
>      limits means a "noisy"/unbounded worker could have service impacting
>      results to other applications. It also prevents the worker
> auto-scaler from
>      working.
>      - Define the entry point for the worker pod - for some reason the
>      original entry point for the worker is not the same as the Airflow Image.
>      We use the entry point to do some dynamic initialisation on the worker.
>      - Ideally, have the ability to supply the full manifest for the
>      worker. This is a proposal over creating an adapter layer on top of
>      Kubernetes which has to be maintained.
> 
> We are keen to support Airflow and also add these features, we'd like to
> understand if you'd be open to contributors?
> 
> We don't want to invest time on the executor if it wouldn't fit in with the
> design you'd like or it never got merged into mainline Airflow. Managing
> forks is a painful process and we are also huge advocates of contributing
> back to OSS.
> 
> FY I spoke to Eamon (on Slack) and he's informed me Daniel from
> Bloomberg did most of the initial work on the executor; I've reached out to
> him directly as well.
> 
> It's great to connect, hopefully, I'll hear back from you.
> 
> Cheers,
> 
> Darren

Mime
View raw message