airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Greene <br...@heisenbergwoodworking.com>
Subject Re: Airflow scheduler process issue
Date Fri, 29 Mar 2019 02:04:39 GMT
My quick guess - both running with local executor?  That runs all af processes in one... so
you’d have 2 schedulers.

It’s perfectly acceptable to run AF across a couple  EC2 instances, but you have to select
which parts of the stack run in which one, and support multiple airflow Configs in some cases.

Sent from a device with less than stellar autocorrect

> On Mar 28, 2019, at 3:11 PM, Adityan MS <adityan@poshmark.com> wrote:
> 
> Hi all,
> 
> We have 2 EC2 instances running airflow 1.10.0 apps.  As per my
> understanding there should be only 1 airflow scheduler service running at a
> given time. However lately we have started to see that mysteriously 2
> scheduler processes are running, which ends up causing a deadlock and our
> Airflow tasks don't run. This ends up giving us an error in the
> `scheduler.logs` - `Executor reports task instance <TaskInstance: <Task
> Name> 2019-03-25 09:00:00+00:00 [queued]> finished (success) although the
> task says its queued. Was the task killed externally?`
> 
> To resolve this I go and manually kill the scheduler process by using `kill
> -9 PID`. The scheduler failover controller takes care of spawning a new
> scheduler process and all tasks resume running. Has anyone faced this
> issue? How can I prevent 2 scheduler processes from coming up, and why do
> they come up since we are not manually issuing a command to bring them up
> in the first place?
> 
> Thanks!

Mime
View raw message