airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-276) List of dags does not refresh in UI for a while
Date Mon, 08 Aug 2016 18:27:20 GMT


ASF subversion and git services commented on AIRFLOW-276:

Commit 9d254a317dd54f555270ca568aff1cd0500e1e53 in incubator-airflow's branch refs/heads/master
from [~xuanji]
[;h=9d254a3 ]

[AIRFLOW-276] Gunicorn rolling restart

- Tell gunicorn to prepend `[ready]` to worker process name once worker is ready (to serve
requests) - in particular this happens after DAGs folder is parsed
- Airflow cli runs gunicorn as a child process instead of `excecvp`-ing over itself
- Airflow cli monitors gunicorn worker processes and restarts them by sending TTIN/TTOU signals
to the gunicorn master process
- Fix bug where `conf.get('webserver', 'workers')` and `conf.get('webserver', 'webserver_worker_timeout')`
were ignored

- Alternatively, does the same
thing but the worker-restart script is provided separately for the user to run

- Start airflow, observe that workers are restarted
- Add new dags to dags folder and check that they show up
- Run `siege` against airflow while server is restarting and confirm that all requests succeed
- Run with configuration set to `batch_size = 0`, `batch_size = 1` and `batch_size = 4`

Closes #1685 from zodiac/xuanji_gunicorn_rolling_restart_2

> List of dags does not refresh in UI for a while
> -----------------------------------------------
>                 Key: AIRFLOW-276
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Li Xuanji
>            Assignee: Li Xuanji
>            Priority: Minor
> After creating a new dag (eg by adding a file to `~/airflow/dags`), the web UI does not
show the new for a while. It only shows it when either
> 1. gunicorn decides to restart the worker process, or
> 2. a scheduler picks up the new dag, adds it to the airflow db, and the web UI notices
it in the db

This message was sent by Atlassian JIRA

View raw message