Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Mon, 26 Sep 2016 21:31:21 +0000 (UTC)
From: "Siddharth Anand (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.12999945.1472112601000.667340.1474925481392@Atlassian.JIRA>
In-Reply-To: <JIRA.12999945.1472112601000@Atlassian.JIRA>
References: <JIRA.12999945.1472112601000@Atlassian.JIRA> <JIRA.12999945.1472112601983@arcas>
Subject: [jira] [Commented] (AIRFLOW-462) Concurrent Scheduler Jobs pushing
 the same task to queue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 26 Sep 2016 21:31:27 -0000


    [ https://issues.apache.org/jira/browse/AIRFLOW-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524228#comment-15524228 ] 

Siddharth Anand commented on AIRFLOW-462:
-----------------------------------------

H!
A few caveats - we ended up turning off dual schedulers because of another issue -- i.e. how we currently log. We also use LocalExecutor, not CeleryExecutors and have never tested with CeleryExecutor. The idea, if using CeleryExecutor, is not to run more than one scheduler. The reason I opted for dual schedulers was to run LocalExecutor on more machines. However, as changes are being made to the scheduler, I don't believe running dual schedulers will be maintained going forward. It's not a first-level feature that most people require. 

-s

> Concurrent Scheduler Jobs pushing the same task to queue
> --------------------------------------------------------
>
>                 Key: AIRFLOW-462
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-462
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: Airflow 1.7.0
>            Reporter: Yogesh
>            Priority: Blocker
>
> Hi,
> We are using airflow version 1.7.0 and we tried to implement high availability for airflow daemons in our production environment.
> Detailed high availability approach:
> -	Airflow running on two different machines with all the daemons(webserver, scheduler, execueor)
> -	Single mysql db repository pointed by two schedulers
> -	Replicated dag files in both the machines
> -       Running Single Rabbitmq Instance as message broker
> While doing so we came across below problem:
> -	A particular task was sent to executor twice (two entries in message queue) by two different schedulers. But, we see only single entry for the task instance in database which is correct.
> We just checked out the code and found below fact:
> -	before sending the task to executor it checks for task state in database and if its not already QUEUED it pushes that task to queue
> issue:
> As there is no locking implemented on the task instance in the database and both the Scheduler jobs are running so close that the second one might check for the status in the db before the first one updates that to QUEUED.
> We are not sure if in recent release this issue have been taken care of.
> Would you please help with some appropriate approach so that the high availability can be achieved.
> Thanks
> Yogesh


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)