airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Ma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-2113) Address missing DagRuns callbacks
Date Thu, 15 Feb 2018 21:12:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alan Ma updated AIRFLOW-2113:
-----------------------------
    Summary: Address missing DagRuns callbacks  (was: DagRuns does not complete callbacks)

> Address missing DagRuns callbacks
> ---------------------------------
>
>                 Key: AIRFLOW-2113
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2113
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Alan Ma
>            Assignee: Alan Ma
>            Priority: Critical
>
> This originally arose from the missing notification from the on_failure and on_success
callback at the dag level. The stack trace is as follows:
> {code:java}
> [2018-02-07 07:00:08,145] \{models.py:2984} DagFileProcessor172 INFO - Executing dag
callback function: <pandora_airflow_plugin.hooks.general_notification_hook
> .GeneralNotifyFailed instance at 0x7fec9d8ad368>
> [2018-02-07 07:00:08,161] \{models.py:168} DagFileProcessor172 INFO - Filling up the
DagBag from /home/charon/.virtualenvs/airflow/airflow_home/dags
> Dag: <DAG: hive_migration_dag>, paused: False
> Dag: <DAG: continuously_refresh_ui>, paused: False
> Dag: <DAG: consul_ttl_check>, paused: False
> Dag: <DAG: example_analytics_dag>, paused: False
> Dag: <DAG: postgres_vacuum>, paused: False
> [2018-02-07 07:00:12,103] \{jobs.py:354} DagFileProcessor172 ERROR - Got an exception!
Propagating...
> Traceback (most recent call last):
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/jobs.py",
line 346, in helper
> pickle_dags)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/utils/db.py",
line 53, in wrapper
> result = func(*args, **kwargs)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/jobs.py",
line 1586, in process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/jobs.py",
line 1175, in _process_dags
> dag_run = self.create_dag_run(dag)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/utils/db.py",
line 53, in wrapper
> result = func(*args, **kwargs)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/jobs.py",
line 747, in create_dag_run
> dag.handle_callback(dr, success=False, reason='dagrun_timeout', session=session)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/utils/db.py",
line 53, in wrapper
> result = func(*args, **kwargs)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/airflow/models.py",
line 2990, in handle_callback
> d = dagrun.dag or DagBag().get_dag(dag_id=dagrun.dag_id)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py",
line 237, in __get__
> return self.impl.get(instance_state(instance), dict_)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py",
line 579, in get
> value = state._load_expired(state, passive)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py",
line 592, in _load_expired
> self.manager.deferred_scalar_loader(self, toload)
> File "/home/charon/.virtualenvs/airflow/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 644, in load_scalar_attributes
> (state_str(state)))
> DetachedInstanceError: Instance <DagRun at 0x7fec9d8c8e90> is not bound to a Session;
attribute refresh operation cannot proceed
> [2018-02-07 07:00:31,003] \{jobs.py:343} DagFileProcessor208 INFO - Started process (PID=7813)
to work on /home/charon/.virtualenvs/airflow/airflow_home/dags/c
> haron-airflow/dags/inapp_vendor_sku_breakdown.py\
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message