airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bolke de Bruin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1493) Fix race condition with airflow run
Date Tue, 08 Aug 2017 13:02:01 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118305#comment-16118305
] 

Bolke de Bruin commented on AIRFLOW-1493:
-----------------------------------------

Why not rely on the OS to verify the task will actually run? I.e. Use signals, these are observable
without the issue your are describing. 

> Fix race condition with airflow run
> -----------------------------------
>
>                 Key: AIRFLOW-1493
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1493
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Alex Guziel
>            Assignee: Alex Guziel
>
> Currently, airflow run spawns a process `airflow run --local` which spawns `airflow run
--raw`.
> Local manages the heartbeat. Raw performs a series of checks, sets the state to running,
runs the task, then sets the state to failed or success. 
> The problem is the heartbeat check on `airflow run --local` has to monitor the state
in the DB, but because the change of state to running happens asynchronously, it must first
observe the state in the DB to be running before it has the power of termination. However,
there is no guarantee that it will observe this state. Thus, we should  move the pre-execution
logic to airflow run --local



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message