airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuri Bendana (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-3415) Imports become null when triggering dagruns in a loop
Date Thu, 29 Nov 2018 18:24:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yuri Bendana updated AIRFLOW-3415:
----------------------------------
    Description: 
When triggering dagruns in a loop, the imported references become null on the second iteration. 
Here is an example [ gist | [https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ].
For the purposes here, you can ignore the DagRunSensor task. On the first iteration the 'sleeper'
dag gets triggered but on the second iteration I see a
{noformat}
TypeError: 'NoneType' object is not callable{noformat}
To workaround this, I have to copy the import (in this case trigger_dag) inside the loop.

Here is the stacktrace:
{code:java}
[2018-11-28 18:23:42,492] {{models.py:1789}} INFO - All retries failed; marking task as FAILED
[2018-11-28 18:23:42,517] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 Traceback
(most recent call last):
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/venv/sgmo/bin/airflow", line 32, in <module>
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     args.func(args)
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     return
f(*args, **kwargs)
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 490, in run
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     _run(args,
dag, ti)
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 406, in _run
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     pool=args.pool,
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in wrapper
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     return
func(*args, **kwargs)
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/models.py", line 1659, in _run_raw_task
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     result
= task_copy.execute(context=context)
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 95,
in execute
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     return_value
= self.execute_callable()
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 100,
in execute_callable
[2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     return
self.python_callable(*self.op_args, **self.op_kwargs)
[2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1   File
"/opt/airflow/dags/sleepersensor.py", line 11, in triggersleeper
[2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1     tdr
= trigger_dag('sleeper')
[2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1 TypeError:
'NoneType' object is not callable
{code}
 

  was:
When triggering dagruns in a loop, the imported references become null on the second iteration. 
Here is an example [ gist | [https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ].
For the purposes here, you can ignore the DagRunSensor task. On the first iteration the 'sleeper'
dag gets triggered but on the second iteration I see a
{noformat}
TypeError: 'NoneType' object is not callable{noformat}
To workaround this, I have to copy the import (in this case trigger_dag) inside the loop.


> Imports become null when triggering dagruns in a loop
> -----------------------------------------------------
>
>                 Key: AIRFLOW-3415
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3415
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DagRun
>    Affects Versions: 1.10.1
>         Environment: CentOS 7
>            Reporter: Yuri Bendana
>            Priority: Minor
>         Attachments: triggerdag_stacktrace.txt
>
>
> When triggering dagruns in a loop, the imported references become null on the second
iteration.  Here is an example [ gist | [https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ].
For the purposes here, you can ignore the DagRunSensor task. On the first iteration the 'sleeper'
dag gets triggered but on the second iteration I see a
> {noformat}
> TypeError: 'NoneType' object is not callable{noformat}
> To workaround this, I have to copy the import (in this case trigger_dag) inside the loop.
> Here is the stacktrace:
> {code:java}
> [2018-11-28 18:23:42,492] {{models.py:1789}} INFO - All retries failed; marking task
as FAILED
> [2018-11-28 18:23:42,517] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
Traceback (most recent call last):
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/venv/sgmo/bin/airflow", line 32, in <module>
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    args.func(args)
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    return f(*args, **kwargs)
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 490, in run
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    _run(args, dag, ti)
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 406, in _run
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    pool=args.pool,
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in wrapper
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    return func(*args, **kwargs)
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/models.py", line 1659, in _run_raw_task
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    result = task_copy.execute(context=context)
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py",
line 95, in execute
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    return_value = self.execute_callable()
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py",
line 100, in execute_callable
> [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    return self.python_callable(*self.op_args, **self.op_kwargs)
> [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
  File "/opt/airflow/dags/sleepersensor.py", line 11, in triggersleeper
> [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
    tdr = trigger_dag('sleeper')
> [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170: Subtask t1
TypeError: 'NoneType' object is not callable
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message