airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Arnold (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1235) Odd behaviour when all gunicorn workers die
Date Mon, 19 Mar 2018 22:28:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405547#comment-16405547
] 

John Arnold commented on AIRFLOW-1235:
--------------------------------------

For reference, here's what it looks linke in my logs too...

```

[2018-03-19 03:21:31,442] \{configuration.py:211} WARNING - section/key [celery/ssl_active]
not found in config
[2018-03-19 03:21:31,442] \{default_celery.py:44} WARNING - Celery Executor will run without
SSL
[2018-03-19 03:21:33,299] \{config.py:26} INFO - Missing required environment vars. Attempting
to load from /etc/celery/conf.d/celery
[2018-03-19 03:21:33,352] \{__init__.py:45} INFO - Using executor CeleryExecutor
[2018-03-19 03:21:33,724] \{models.py:196} INFO - Filling up the DagBag from /var/lib/airflow/venv/lib/python3.6/site-packages/net_task_contrib/airflow/dags
[2018-03-19 03:21:36 +0000] [125724] [INFO] Handling signal: ttou
[2018-03-19 03:21:36 +0000] [55375] [INFO] Worker exiting (pid: 55375)
[2018-03-19 03:22:07 +0000] [125724] [INFO] Handling signal: ttin
[2018-03-19 03:22:07 +0000] [59230] [INFO] Booting worker with pid: 59230
[2018-03-19 03:22:07,621] \{configuration.py:211} WARNING - section/key [celery/ssl_active]
not found in config
[2018-03-19 03:22:07,621] \{default_celery.py:44} WARNING - Celery Executor will run without
SSL
[2018-03-19 03:22:08,604] \{config.py:26} INFO - Missing required environment vars. Attempting
to load from /etc/celery/conf.d/celery
[2018-03-19 03:22:08,648] \{__init__.py:45} INFO - Using executor CeleryExecutor
[2018-03-19 03:22:08,979] \{models.py:196} INFO - Filling up the DagBag from /var/lib/airflow/venv/lib/python3.6/site-packages/net_task_contrib/airflow/dags
[2018-03-19 03:22:09 +0000] [59230] [ERROR] Exception in worker process
Traceback (most recent call last):
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1124, in
_do_get
 return self._pool.get(wait, self._timeout)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/queue.py", line 145,
in get
 raise Empty
sqlalchemy.util.queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
2147, in _wrap_pool_connect
 return fn()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 387, in
connect
 return _ConnectionFairy._checkout(self)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 768, in
_checkout
 fairy = _ConnectionRecord.checkout(pool)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 516, in
checkout
 rec = pool._do_get()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1140, in
_do_get
 self._dec_overflow()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
 compat.reraise(exc_type, exc_value, exc_tb)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line
187, in reraise
 raise value
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1137, in
_do_get
 return self._create_connection()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 333, in
_create_connection
 return _ConnectionRecord(self)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 461, in
__init__
 self.__connect(first_connect_check=True)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 651, in
__connect
 connection = pool._invoke_creator(self)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
 return dialect.connect(*cargs, **cparams)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
393, in connect
 return self.dbapi.connect(*cargs, **cparams)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/psycopg2/__init__.py", line 130,
in connect
 conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: SSL SYSCALL error: EOF detected


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 578, in
spawn_worker
 worker.init_process()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 126,
in init_process
 self.load_wsgi()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 135,
in load_wsgi
 self.wsgi = self.app.wsgi()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in
wsgi
 self.callable = self.load()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 65,
in load
 return self.load_wsgiapp()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52,
in load_wsgiapp
 return util.import_app(self.app_uri)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/util.py", line 364, in import_app
 app = eval(obj, vars(mod))
 File "<string>", line 1, in <module>
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/app.py", line 177, in
cached_app
 app = create_app(config, testing)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/app.py", line 63, in
create_app
 from airflow.www import views
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/views.py", line 2099,
in <module>
 class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/views.py", line 2190,
in ChartModelView
 'conn_id': _connection_ids()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/utils/db.py", line 69, in
wrapper
 return func(*args, **kwargs)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/views.py", line 2094,
in _connection_ids
 .group_by(models.Connection.conn_id)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2855,
in __iter__
 return self._execute_and_instances(context)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2876,
in _execute_and_instances
 close_with_result=True)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2885,
in _get_bind_args
 **kw
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2867,
in _connection_from_session
 conn = self.session.connection(**kw)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line
1019, in connection
 execution_options=execution_options)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line
1024, in _connection_for_bind
 engine, execution_options)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line
409, in _connection_for_bind
 conn = bind.contextual_connect()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
2112, in contextual_connect
 self._wrap_pool_connect(self.pool.connect, None),
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
2151, in _wrap_pool_connect
 e, dialect, self)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
1465, in _handle_dbapi_exception_noconnection
 exc_info
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line
203, in raise_from_cause
 reraise(type(exception), exception, tb=exc_tb, cause=cause)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line
186, in reraise
 raise value.with_traceback(tb)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
2147, in _wrap_pool_connect
 return fn()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 387, in
connect
 return _ConnectionFairy._checkout(self)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 768, in
_checkout
 fairy = _ConnectionRecord.checkout(pool)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 516, in
checkout
 rec = pool._do_get()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1140, in
_do_get
 self._dec_overflow()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
 compat.reraise(exc_type, exc_value, exc_tb)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line
187, in reraise
 raise value
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1137, in
_do_get
 return self._create_connection()
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 333, in
_create_connection
 return _ConnectionRecord(self)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 461, in
__init__
 self.__connect(first_connect_check=True)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 651, in
__connect
 connection = pool._invoke_creator(self)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
 return dialect.connect(*cargs, **cparams)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
393, in connect
 return self.dbapi.connect(*cargs, **cparams)
 File "/var/lib/airflow/venv/lib/python3.6/site-packages/psycopg2/__init__.py", line 130,
in connect
 conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected

[2018-03-19 03:22:09 +0000] [59230] [INFO] Worker exiting (pid: 59230)
[2018-03-19 03:22:09 +0000] [57686] [INFO] Worker exiting (pid: 57686)
[2018-03-19 03:22:09 +0000] [58454] [INFO] Worker exiting (pid: 58454)
[2018-03-19 03:22:09 +0000] [56148] [INFO] Worker exiting (pid: 56148)
[2018-03-19 03:22:09 +0000] [56923] [INFO] Worker exiting (pid: 56923)
[2018-03-19 03:22:10 +0000] [125724] [INFO] Shutting down: Master
[2018-03-19 03:22:10 +0000] [125724] [INFO] Reason: Worker failed to boot.
[2018-03-19 03:22:10,322] \{cli.py:686} ERROR - [0 / 0] some workers seem to have died and
gunicorndid not restart them as expected

> Odd behaviour when all gunicorn workers die
> -------------------------------------------
>
>                 Key: AIRFLOW-1235
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1235
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: webserver
>    Affects Versions: 1.8.0
>            Reporter: Erik Forsberg
>            Assignee: Kengo Seki
>            Priority: Major
>
> The webserver has sometimes stopped responding to port 443, and today I found the issue
- I had a misconfigured resolv.conf that made it unable to talk to my postgresql. This was
the root cause, but the way airflow webserver behaved was a bit odd.
> It seems that when all gunicorn workers failed to start, the gunicorn master shut down.
However, the main process (the one that starts gunicorn master) did not shut down, so there
was no way of detecting the failed status of webserver from e.g. systemd or init script.
> Full traceback leading to stale webserver process:
> {noformat}
> May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] [23794] [ERROR]
Exception in worker process:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above exception, another
exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory,
**kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could not translate
host name "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the direct cause
of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
line 67, in wsgi
> May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 65, in load
> May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 52, in load_wsgiapp
> May 21 09:51:57 airmaster01 airflow[26451]: return util.import_app(self.app_uri)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py",
line 366, in import_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
line 161, in cached_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
line 60, in create_app
> May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 1975, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: class ChartModelView(wwwutils.DataProfilingMixin,
AirflowModelView):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 2057, in ChartModelView
> May 21 09:51:57 airmaster01 airflow[26451]: .group_by(models.Connection.conn_id)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2855, in __iter__
> May 21 09:51:57 airmaster01 airflow[26451]: return self._execute_and_instances(context)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2876, in _execute_and_instances
> May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2885, in _get_bind_args
> May 21 09:51:57 airmaster01 airflow[26451]: **kw
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2867, in _connection_from_session
> May 21 09:51:57 airmaster01 airflow[26451]: conn = self.session.connection(**kw)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 966, in connection
> May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 971, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: closed.  This flag only has an effect if
this :class:`.Session` is
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 403, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2112, in contextual_connect
> May 21 09:51:57 airmaster01 airflow[26451]: self._wrap_pool_connect(self.pool.connect,
None),
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2151, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:57 airmaster01 airflow[26451]: exc_info
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
> May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception), exception, tb=exc_tb,
cause=cause)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 186, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory,
**kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
could not translate host name "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above exception, another
exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory,
**kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could not translate
host name "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the direct cause
of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
line 67, in wsgi
> May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 65, in load
> May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 52, in load_wsgiapp
> May 21 09:51:58 airmaster01 airflow[26451]: return util.import_app(self.app_uri)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py",
line 366, in import_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
line 161, in cached_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
line 60, in create_app
> May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 1975, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: class ChartModelView(wwwutils.DataProfilingMixin,
AirflowModelView):
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 2057, in ChartModelView
> May 21 09:51:58 airmaster01 airflow[26451]: .group_by(models.Connection.conn_id)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2855, in __iter__
> May 21 09:51:58 airmaster01 airflow[26451]: return self._execute_and_instances(context)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2876, in _execute_and_instances
> May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2885, in _get_bind_args
> May 21 09:51:58 airmaster01 airflow[26451]: **kw
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2867, in _connection_from_session
> May 21 09:51:58 airmaster01 airflow[26451]: conn = self.session.connection(**kw)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 966, in connection
> May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 971, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: closed.  This flag only has an effect if
this :class:`.Session` is
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 403, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2112, in contextual_connect
> May 21 09:51:58 airmaster01 airflow[26451]: self._wrap_pool_connect(self.pool.connect,
None),
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2151, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:58 airmaster01 airflow[26451]: exc_info
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
> May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception), exception, tb=exc_tb,
cause=cause)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 186, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: return fn()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 387, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 766, in _checkout
> May 21 09:51:58 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 516, in checkout
> May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1138, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
> May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 1135, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 461, in __init__
> May 21 09:51:58 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
line 651, in __connect
> May 21 09:51:58 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory,
**kwasync)
> May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
could not translate host name "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] [23794] [INFO]
Worker exiting (pid: 23794)
> May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23561] [INFO]
Worker exiting (pid: 23561)
> May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23759] [INFO]
Worker exiting (pid: 23759)
> May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23620] [INFO]
Worker exiting (pid: 23620)
> May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23689] [INFO]
Worker exiting (pid: 23689)
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] [26464] [INFO]
Shutting down: Master
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] [26464] [INFO]
Reason: Worker failed to boot.
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Mime
View raw message