airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renaud Grisoni <renaud.gris...@gmail.com>
Subject Airflow local scheduler bugs but stays running
Date Mon, 07 Nov 2016 10:54:13 GMT
Hi all,

I use Airflow v1.7.1.3 with the local scheduler and I encounter a problem
with the scheduler :
For some reason, the airflow database is no more accessible so the
scheduler display the OperationalError below. My problem is the scheduler
does not kill itself after this error, it is running but it does not run
any DAG any more. I cannot automatically restart it with Supervisor because
its process is always displayed as runnning. Each time I have a network
error, Airflow display this error and enters in this "zombie" mode, and my
DAG are not processed.

Have you heard about this problem, any suggestions?



29/09/2016 21:09:53Traceback (most recent call last):
29/09/2016 21:09:53  File "/usr/bin/airflow", line 15, in <module>
29/09/2016 21:09:53    args.func(args)
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py",
line 455, in scheduler
29/09/2016 21:09:53    job.run()
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packages/airflow/jobs.py",
line 173, in run
29/09/2016 21:09:53    self._execute()
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packages/airflow/jobs.py",
line 712, in _execute
29/09/2016 21:09:53    paused_dag_ids = dagbag.paused_dags()
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packages/airflow/models.py",
line 429, in paused_dags
29/09/2016 21:09:53    DagModel.is_paused == True)]
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2761, in __iter__
29/09/2016 21:09:53    return self._execute_and_instances(context)
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2774, in _execute_and_instances
29/09/2016 21:09:53    close_with_result=True)
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2765, in _connection_from_session
29/09/2016 21:09:53    **kw)
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/orm/session.py", line 893, in connection
29/09/2016 21:09:53    execution_options=execution_options)
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
29/09/2016 21:09:53    engine, execution_options)
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
29/09/2016 21:09:53    conn = bind.contextual_connect()
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/engine/base.py", line 2039, in contextual_connect
29/09/2016 21:09:53    self._wrap_pool_connect(self.pool.connect, None),
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
29/09/2016 21:09:53    e, dialect, self)
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconn
ection
29/09/2016 21:09:53    exc_info
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/util/compat.py", line 202, in raise_from_cause
29/09/2016 21:09:53    reraise(type(exception), exception, tb=exc_tb,
cause=cause)
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
29/09/2016 21:09:53    return fn()
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 376, in connect
29/09/2016 21:09:53    return _ConnectionFairy._checkout(self)
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 713, in _checkout
29/09/2016 21:09:53    fairy = _ConnectionRecord.checkout(pool)
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 485, in checkout
29/09/2016 21:09:53    rec.checkin()
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/util/langhelpers.py", line 60, in __exit__
29/09/2016 21:09:53    compat.reraise(exc_type, exc_value, exc_tb)
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 482, in checkout
29/09/2016 21:09:53    dbapi_connection = rec.get_connection()
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 563, in get_connection
29/09/2016 21:09:53    self.connection = self.__connect()
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/sqlalchemy/pool.py",
line 607, in __connect
29/09/2016 21:09:53    connection = self.__pool._invoke_creator(self)
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/engine/strategies.py", line 97, in connect
29/09/2016 21:09:53    return dialect.connect(*cargs, **cparams)
29/09/2016 21:09:53  File "/usr/lib/python2.7/site-packa
ges/sqlalchemy/engine/default.py", line 385, in connect
29/09/2016 21:09:53    return self.dbapi.connect(*cargs, **cparams)
29/09/2016 21:09:53  File
"/usr/lib/python2.7/site-packages/psycopg2/__init__.py",
line 164, in connect
29/09/2016 21:09:53    conn = _connect(dsn,
connection_factory=connection_factory,
async=async)
29/09/2016 21:09:53sqlalchemy.exc.OperationalError:
(psycopg2.OperationalError) could not translate host name "db-airflow" to
address: Name does not resolve

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message