airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kengo Seki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1235) Odd behaviour when all gunicorn workers die
Date Wed, 24 May 2017 13:32:04 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022892#comment-16022892
] 

Kengo Seki commented on AIRFLOW-1235:
-------------------------------------

I could reproduce this issue as follows:

1. Run webserver in foreground
2. Kill gunicorn master

{code}
$ ps aux | grep airflow | grep -v grep
sekikn    2929  4.5  9.3 241204 71172 pts/1    S+   09:12   0:02 /home/sekikn/.virtualenvs/a/bin/python2
/home/sekikn/.virtualenvs/a/bin/airflow webserver
sekikn    2935  1.9  6.1 115868 46676 pts/1    S+   09:12   0:00 gunicorn: master [airflow-webserver]
sekikn    2943  1.3  8.8 242144 67300 pts/1    Sl+  09:12   0:00 [ready] gunicorn: worker
[airflow-webserver]
sekikn    2944  1.3  8.8 242040 67312 pts/1    Sl+  09:12   0:00 [ready] gunicorn: worker
[airflow-webserver]
sekikn    2945  1.3  8.8 242052 67320 pts/1    Sl+  09:12   0:00 [ready] gunicorn: worker
[airflow-webserver]
sekikn    2952  6.0  8.8 242056 67196 pts/1    Sl+  09:13   0:00 [ready] gunicorn: worker
[airflow-webserver]
$ kill 2935
{code}

3. Then gunicorn master remains as a zombie and webserver gets stuck

{code}
$ ps aux | grep airflow | grep -v grep
sekikn    2929 10.5  9.3 241204 71220 pts/1    S+   09:12   0:22 /home/sekikn/.virtualenvs/a/bin/python2
/home/sekikn/.virtualenvs/a/bin/airflow webserver
$ ps 2935
  PID TTY      STAT   TIME COMMAND
 2935 pts/1    Z+     0:00 [gunicorn: maste] <defunct>
{code}

At step 3, the following message is output to the log:

{code}
[2017-05-24 09:13:52,092] [2929] {cli.py:671} DEBUG - [4 / 4] doing a refresh of 1 workers
{code}

So I think webserver waits for workers infinitely (but they will never start) at line 679.

{code}
 668     def start_refresh(gunicorn_master_proc):
 669         batch_size = conf.getint('webserver', 'worker_refresh_batch_size')
 670         logging.debug('%s doing a refresh of %s workers',
 671                       state, batch_size)
 672         sys.stdout.flush()
 673         sys.stderr.flush()
 674 
 675         excess = 0
 676         for _ in range(batch_size):
 677             gunicorn_master_proc.send_signal(signal.SIGTTIN)
 678             excess += 1
 679             wait_until_true(lambda: num_workers_expected + excess ==
 680                             get_num_workers_running(gunicorn_master_proc))
{code}

> Odd behaviour when all gunicorn workers die
> -------------------------------------------
>
>                 Key: AIRFLOW-1235
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1235
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: webserver
>    Affects Versions: 1.8.0
>            Reporter: Erik Forsberg
>
> The webserver has sometimes stopped responding to port 443, and today I found the issue
- I had a misconfigured resolv.conf that made it unable to talk to my postgresql. This was
the root cause, but the way airflow webserver behaved was a bit odd.
> It seems that when all gunicorn workers failed to start, the gunicorn master shut down.
However, the main process (the one that starts gunicorn master) did not shut down, so there
was no way of detecting the failed status of webserver from e.g. systemd or init script.
> Full traceback leading to stale webserver process:
> {noformat}
> May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] [23794] [ERROR]
Exception in worker process:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above exception, another
exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory,
**kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could not translate
host name "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the direct cause
of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
line 67, in wsgi
> May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 65, in load
> May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 52, in load_wsgiapp
> May 21 09:51:57 airmaster01 airflow[26451]: return util.import_app(self.app_uri)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py",
line 366, in import_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
line 161, in cached_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
line 60, in create_app
> May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 1975, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: class ChartModelView(wwwutils.DataProfilingMixin,
AirflowModelView):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 2057, in ChartModelView
> May 21 09:51:57 airmaster01 airflow[26451]: .group_by(models.Connection.conn_id)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2855, in __iter__
> May 21 09:51:57 airmaster01 airflow[26451]: return self._execute_and_instances(context)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2876, in _execute_and_instances
> May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2885, in _get_bind_args
> May 21 09:51:57 airmaster01 airflow[26451]: **kw
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2867, in _connection_from_session
> May 21 09:51:57 airmaster01 airflow[26451]: conn = self.session.connection(**kw)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 966, in connection
> May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 971, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: closed.  This flag only has an effect if
this :class:`.Session` is
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 403, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2112, in contextual_connect
> May 21 09:51:57 airmaster01 airflow[26451]: self._wrap_pool_connect(self.pool.connect,
None),
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2151, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:57 airmaster01 airflow[26451]: exc_info
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
> May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception), exception, tb=exc_tb,
cause=cause)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 186, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory,
**kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
could not translate host name "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above exception, another
exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory,
**kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could not translate
host name "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the direct cause
of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
line 67, in wsgi
> May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 65, in load
> May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 52, in load_wsgiapp
> May 21 09:51:58 airmaster01 airflow[26451]: return util.import_app(self.app_uri)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py",
line 366, in import_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
line 161, in cached_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
line 60, in create_app
> May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 1975, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: class ChartModelView(wwwutils.DataProfilingMixin,
AirflowModelView):
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 2057, in ChartModelView
> May 21 09:51:58 airmaster01 airflow[26451]: .group_by(models.Connection.conn_id)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2855, in __iter__
> May 21 09:51:58 airmaster01 airflow[26451]: return self._execute_and_instances(context)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2876, in _execute_and_instances
> May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2885, in _get_bind_args
> May 21 09:51:58 airmaster01 airflow[26451]: **kw
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2867, in _connection_from_session
> May 21 09:51:58 airmaster01 airflow[26451]: conn = self.session.connection(**kw)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 966, in connection
> May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 971, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: closed.  This flag only has an effect if
this :class:`.Session` is
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 403, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2112, in contextual_connect
> May 21 09:51:58 airmaster01 airflow[26451]: self._wrap_pool_connect(self.pool.connect,
None),
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2151, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:58 airmaster01 airflow[26451]: exc_info
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
> May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception), exception, tb=exc_tb,
cause=cause)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 186, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: return fn()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 387, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 766, in _checkout
> May 21 09:51:58 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 516, in checkout
> May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1138, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
> May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1135, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 461, in __init__
> May 21 09:51:58 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 651, in __connect
> May 21 09:51:58 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory,
**kwasync)
> May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
could not translate host name "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] [23794] [INFO]
Worker exiting (pid: 23794)
> May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23561] [INFO]
Worker exiting (pid: 23561)
> May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23759] [INFO]
Worker exiting (pid: 23759)
> May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23620] [INFO]
Worker exiting (pid: 23620)
> May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23689] [INFO]
Worker exiting (pid: 23689)
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] [26464] [INFO]
Shutting down: Master
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] [26464] [INFO]
Reason: Worker failed to boot.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message