airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Barker (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (AIRFLOW-1885) Exception when polling ready workers and a gunicorn worker becomes a zombie
Date Tue, 05 Dec 2017 08:24:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

John Barker reassigned AIRFLOW-1885:
------------------------------------

    Assignee: John Barker

> Exception when polling ready workers and a gunicorn worker becomes a zombie
> ---------------------------------------------------------------------------
>
>                 Key: AIRFLOW-1885
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1885
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: Airflow 1.8
>            Reporter: John Barker
>            Assignee: John Barker
>
> If one of the gunicorn workers happens to become a zombie between {{children()}} and
{{cmdline()}} calls to psutil in {{get_num_ready_workers_running}} will raise an IndexError:
> {code}
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 28, in <module>
>     args.func(args)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 803, in webserver
>     restart_workers(gunicorn_master_proc, num_workers)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 687, in restart_workers
>     num_ready_workers_running = get_num_ready_workers_running(gunicorn_master_proc)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 663, in get_num_ready_workers_running
>     proc for proc in workers
>   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 664, in <listcomp>
>     if settings.GUNICORN_WORKER_READY_PREFIX in proc.cmdline()[0]
> IndexError: list index out of range
> {code}
> In version 4.2 of psutil, {{cmdline}} can return an empty array if the process is zombied:
https://github.com/giampaolo/psutil/blob/release-4.2.0/psutil/_pslinux.py#L1007 so ensure
that an array is actually returned from {{cmdline}} before doing the {{in}} check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message