airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feng Lu <fen...@google.com.INVALID>
Subject Deprecating Run task from Airflow webUI
Date Fri, 29 Jun 2018 23:54:39 GMT
Hi all,

Please take a look at our proposal to deprecate Run task in Airflow webUI.

*What?*
Deprecate Run task support in the Airflow webUI and make it a no-op for
now.

[image: xGAcOrcLJE4.png]
‚Äč
*Why?*

   1. It only works with CeleryExecutor
   <https://github.com/apache/incubator-airflow/blob/master/airflow/www/views.py#L1001-L1003>
   and renders an inconsistent experience for other types of Executors.
   2. It requires Airflow webserver to have direct connection with the
   message backend of CeleryExecutor, and opens more vulnerability in the
   system. In many cases, users may want to restrict access to the celery
   messaging backend as much as possible.

*Mitigation:*
This Run task feature is mainly for the purpose of re-executing of a
previously running task which got stuck in running and deleted manually.
It's currently a two step process:
1. Navigate to the task instance view page and delete the running task
that's stuck
2. Go back to DAG/task view and click "Run"

We proposed to combine the two steps, after a running task is deleted, the
Airflow scheduler will automatically re-schedule (which it does today) and
re-queue the task (there's a bug that needs to be fixed).

*Fix:*
The scheduler currently doesn't not automatically re-queue the task despite
the task instance has changed from running to scheduled state. The
heartbeat check incorrectly returns a success in this case. The root cause
is that LocalTaskJob doesn't set the job state to failed (details
<https://github.com/apache/incubator-airflow/blob/master/airflow/jobs.py#L2674-L2681>)
when a running task is externally deleted and confuses the heartbeat check
<https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L443>
.
Once this is fixed, a killed running task instance will be
auto-scheduled/enqueued for execution, verified locally.

Thank you.

Feng

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message