airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-160) Parse DAG files through child processes
Date Wed, 13 Sep 2017 11:27:00 GMT


ASF subversion and git services commented on AIRFLOW-160:

Commit 028b3b88ff4f191c78bf1d9c41bf43a792f640ff in incubator-airflow's branch refs/heads/master
from [~ashb]
[;h=028b3b8 ]

[AIRFLOW-1606][Airflow-1606][AIRFLOW-1605][AIRFLOW-160] DAG.sync_to_db is now a normal method

Previously it was a static method that took as
it's first argument a
DAG, which really meant it wasn't truly a static

To avoid reversing the parameter order I have
given sensible defaults
from the one and only use in the rest of the code

Also remove documented "sync_to_db" parameter on
DagBag that no longer
exists -- this doc string refers to a parameter
that was removed in

Closes #2605 from ashb/AIRFLOW-1606-db-sync_to_db-

> Parse DAG files through child processes
> ---------------------------------------
>                 Key: AIRFLOW-160
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Paul Yang
>            Assignee: Paul Yang
> Currently, the Airflow scheduler parses all user DAG files in the same process as the
scheduler itself. We've seen issues in production where bad DAG files cause scheduler to fail.
A simple example is if the user script calls `sys.exit(1)`, the scheduler will exit as well.
We've also seen an unusual case where modules loaded by the user DAG affect operation of the
scheduler. For better uptime, the scheduler should be resistant to these problematic user
> The proposed solution is to parse and schedule user DAGs through child processes. This
way, the main scheduler process is more isolated from bad DAGs. There's a side benefit as
well - since parsing is distributed among multiple processes, it's possible to parse the DAG
files more frequently, reducing the latency between when a DAG is modified and when the changes
are picked up.
> Another issue right now is that all DAGs must be scheduled before any tasks are sent
to the executor. This means that the frequency of task scheduling is limited by the slowest
DAG to schedule. The changes needed for scheduling DAGs through child processes will also
make it easy to decouple this process and allow tasks to be scheduled and sent to the executor
in a more independent fashion. This way, overall scheduling won't be held back by a slow DAG.

This message was sent by Atlassian JIRA

View raw message