airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adityan MS <adit...@poshmark.com>
Subject Airflow scheduler process issue
Date Thu, 28 Mar 2019 20:11:16 GMT
Hi all,

We have 2 EC2 instances running airflow 1.10.0 apps.  As per my
understanding there should be only 1 airflow scheduler service running at a
given time. However lately we have started to see that mysteriously 2
scheduler processes are running, which ends up causing a deadlock and our
Airflow tasks don't run. This ends up giving us an error in the
`scheduler.logs` - `Executor reports task instance <TaskInstance: <Task
Name> 2019-03-25 09:00:00+00:00 [queued]> finished (success) although the
task says its queued. Was the task killed externally?`

To resolve this I go and manually kill the scheduler process by using `kill
-9 PID`. The scheduler failover controller takes care of spawning a new
scheduler process and all tasks resume running. Has anyone faced this
issue? How can I prevent 2 scheduler processes from coming up, and why do
they come up since we are not manually issuing a command to bring them up
in the first place?

Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message