airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darren Haken <darrenha...@gmail.com>
Subject Contributing to the Kubernetes Exeuctor
Date Wed, 06 Feb 2019 09:41:50 GMT
Hi,

My name is Darren and I'm head of data at Auto Trader UK. Auto Trader is
the largest automotive marketplace, one of the largest sites in the UK
(based on traffic).

We have been using Airflow for about 18 months with the LocalExecutor and
now the K8 Executor is out we've been migrating towards that.

FYI we are heavy users of Kubernetes:

   - https://cloud.google.com/customers/auto-trader-uk/
   -
   https://cloudplatform.googleblog.com/2018/07/istio-reaches-1-0-ready-for-prod.html

Whilst working on the migration, we've found there are some improvements
which would be good to make to the executor:

   - Support longer DAG names - we've opened a PR for this and are working
   on it already. https://github.com/apache/airflow/pull/4636
   - Improve the executor from a Kubernetes Operator perspective.
      - Define requests and limits for Kubernetes for the workers. We
      operate a multi-tenant cluster across teams and not being able to define
      limits means a "noisy"/unbounded worker could have service impacting
      results to other applications. It also prevents the worker
auto-scaler from
      working.
      - Define the entry point for the worker pod - for some reason the
      original entry point for the worker is not the same as the Airflow Image.
      We use the entry point to do some dynamic initialisation on the worker.
      - Ideally, have the ability to supply the full manifest for the
      worker. This is a proposal over creating an adapter layer on top of
      Kubernetes which has to be maintained.

We are keen to support Airflow and also add these features, we'd like to
understand if you'd be open to contributors?

We don't want to invest time on the executor if it wouldn't fit in with the
design you'd like or it never got merged into mainline Airflow. Managing
forks is a painful process and we are also huge advocates of contributing
back to OSS.

FY I spoke to Eamon (on Slack) and he's informed me Daniel from
Bloomberg did most of the initial work on the executor; I've reached out to
him directly as well.

It's great to connect, hopefully, I'll hear back from you.

Cheers,

Darren

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message