airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Rigney <sjrig...@gmail.com>
Subject Concurrent schedulers
Date Mon, 22 May 2017 05:52:47 GMT
Hi,

We're running airflow in production, but for reliability (n.b. not
performance) we'd like to confirm if it is safe to spawn multiple instances
of the scheduler overlapping in time (otherwise we may need to put more
effort into assuring two copies aren't ever spawned at once in our
environment).


It seems this officially wasn't a supported configuration back in 2015 (
https://groups.google.com/d/msg/airbnb_airflow/-1wKa3OcwME/uATa8y3YDAAJ ),
but has sufficient intra-airflow locking been added that it is now safe to
start up two temporally overlapping instances of the scheduler for the same
airflow system?


Or should we hack in a "global scheduler lock" - we're not looking for
increased performance by scheduler parallelism, just that if we ever fire
up two instances of the scheduler nothing terrible happens?


Stephen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message