airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuri Bendana (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-166) Webserver times out using systemd script
Date Wed, 08 Jun 2016 01:49:20 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319846#comment-15319846
] 

Yuri Bendana commented on AIRFLOW-166:
--------------------------------------

The airflow-webserver.service does pass in the --pid flag

{{ExecStart=/bin/airflow webserver --pid /run/airflow/webserver.pid}}

 and I verified that the location is writable by the airflow user.  When I run {{airflow webserver}}
from the command line it does create the pid file.  I tried passing in the --log, --stdout,
and --stderr but it didn't have any effect.  I then took a look at cli.py and noticed that
they aren't used in the call to gunicorn.  gunicorn doesn't have --stdout and --stderr flags
but it does have an --error-logfile flag.  I tried adding it to the run_args in cli.py but
I then get an 'ImportError: Import by filename is not supported.'  However, I'm able to successfully
run it from the command line although I had to remove the parenthesis after cached_app:

{{gunicorn -w 4 -k sync -t 120 -b 0.0.0.0:8080 -n airflow-webserver -p /opt/airflow/airflow-webserver.pid
--error-logfile /opt/airflow/airflow-webserver.log airflow.www.app:cached_app}}

Any reason why the gunicorn error logfile is not set in cli.py?

> Webserver times out using systemd script
> ----------------------------------------
>
>                 Key: AIRFLOW-166
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-166
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: Airflow 1.7.1
>         Environment: CentOS 7
>            Reporter: Yuri Bendana
>
> I just upgraded to 1.7.1.2 from 1.6 and I'm having a problem starting the webserver using
the systemd script.  This used to work fine.  The issue is that it starts and then just hangs,
no error is reported and it finally times out after about a minute.  I tried starting it from
the command line and it works fine without timing out.  I also ran it in daemon mode with
-D and again it seems to be fine.  Any thoughts on how to debug this?
> Here's the log output:
> {code}
> May 23 16:27:50 ybendana-linux systemd: Starting Airflow webserver daemon...
> May 23 16:27:51 ybendana-linux airflow: [2016-05-23 16:27:51,444] {__init__.py:36} INFO
- Using executor LocalExecutor
> May 23 16:27:53 ybendana-linux airflow: ____________       _____________
> May 23 16:27:53 ybendana-linux airflow: ____    |__( )_________  __/__  /________   
  __
> May 23 16:27:53 ybendana-linux airflow: ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /|
/ /
> May 23 16:27:53 ybendana-linux airflow: ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/
|/ /
> May 23 16:27:53 ybendana-linux airflow: _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
> May 23 16:27:53 ybendana-linux airflow: [2016-05-23 16:27:53,446] {models.py:154} INFO
- Filling up the DagBag from /opt/airflow/dags
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [15960] [INFO] Starting
gunicorn 19.3.0
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [15960] [INFO] Listening
at: http://0.0.0.0:8080 (15960)
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [15960] [INFO] Using
worker: sync
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [16067] [INFO] Booting
worker with pid: 16067
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [16069] [INFO] Booting
worker with pid: 16069
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [16070] [INFO] Booting
worker with pid: 16070
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [16071] [INFO] Booting
worker with pid: 16071
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55,876] {__init__.py:36} INFO
- Using executor LocalExecutor
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55,950] {__init__.py:36} INFO
- Using executor LocalExecutor
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55,972] {__init__.py:36} INFO
- Using executor LocalExecutor
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55,997] {__init__.py:36} INFO
- Using executor LocalExecutor
> May 23 16:27:57 ybendana-linux airflow: [2016-05-23 16:27:57,885] {models.py:154} INFO
- Filling up the DagBag from /opt/airflow/dags
> May 23 16:27:57 ybendana-linux airflow: [2016-05-23 16:27:57,951] {models.py:154} INFO
- Filling up the DagBag from /opt/airflow/dags
> May 23 16:27:57 ybendana-linux airflow: [2016-05-23 16:27:57,983] {models.py:154} INFO
- Filling up the DagBag from /opt/airflow/dags
> May 23 16:27:58 ybendana-linux airflow: [2016-05-23 16:27:58,014] {models.py:154} INFO
- Filling up the DagBag from /opt/airflow/dags
> May 23 16:29:20 ybendana-linux systemd: airflow-webserver.service start operation timed
out. Terminating.
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [16070] [INFO] Worker
exiting (pid: 16070)
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [15960] [INFO] Handling
signal: term
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [16071] [INFO] Worker
exiting (pid: 16071)
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [16069] [INFO] Worker
exiting (pid: 16069)
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [16067] [INFO] Worker
exiting (pid: 16067)
> May 23 16:29:21 ybendana-linux airflow: [2016-05-23 16:29:21 +0000] [15960] [INFO] Shutting
down: Master
> May 23 16:29:21 ybendana-linux systemd: Failed to start Airflow webserver daemon.
> May 23 16:29:21 ybendana-linux systemd: Unit airflow-webserver.service entered failed
state.
> May 23 16:29:21 ybendana-linux systemd: airflow-webserver.service failed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message