airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Toonstra <>
Subject Re: Single Airflow Instance Vs Multiple Airflow Instance
Date Wed, 06 Jun 2018 16:01:43 GMT
We are using two cluster instances. One cluster is for the engineering
teams that are in the "tech" wing and which rigorously follow
tech principles, the other instance is for use by business analysts and
more ad-hoc, experimental work, who do not necessarily follow the
principles. We have a nomad engineer helping out the ad-hoc cluster,
setting it up, connecting it to all systems and resolving
programming questions. All clusters are fully puppetized, so we reuse
configs and ways how things are configured, plus have a common
"platform code" package that is reused across both clusters.


On Wed, Jun 6, 2018 at 5:50 PM, James Meickle <>

> An important consideration here is that there are several settings that are
> cluster-wide. In particular, cluster-wide concurrency settings could result
> in Team B's DAG refusing to schedule based on an error in Team A's DAG.
> Do your teams follow similar practices in how eagerly they ship code, or
> have similar SLAs for resolving issues? If so, you are probably fine using
> co-tenancy. If not, you should probably talk about it first to make sure
> the teams are okay with co-tenancy.
> On Wed, Jun 6, 2018 at 11:24 AM, <
>> wrote:
> > Hi Everyone,
> >
> > We have been experimenting with airflow for about 6 months now.
> > We are planning to have multiple departments to use it. Since we don't
> > have any internal experience with Airflow we are wondering if single
> > instance per department is more suited than single instance with
> > multi-tenancy? We have been aware about the upcoming release of airflow
> > 1.10 and changes that will be made to the RBAC which will be more suited
> > for multi-tenancy.
> >
> > Any advice on this ? Any tips could be helpful to us.
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message