airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxime Beauchemin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-860) Circular module dependency prevents loading of custom executor plugin
Date Tue, 13 Jun 2017 15:39:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Maxime Beauchemin updated AIRFLOW-860:
--------------------------------------
    Fix Version/s:     (was: 1.8.2)
                   1.9.0

> Circular module dependency prevents loading of custom executor plugin
> ---------------------------------------------------------------------
>
>                 Key: AIRFLOW-860
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-860
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: executor, plugins
>    Affects Versions: Airflow 1.8
>         Environment: Linux RHEL 7.6, Python 2.7.13
>            Reporter: Nick Ivanov
>             Fix For: 1.9.0
>
>
> Custom plugins cannot be loaded, which prevents airflow from running, due to apparent
cyclic dependency in plugins_manager called in {{executors.\_\_init\_\_}} -- the top-level
{{\_\_init\_\_}} attempts to load the default executor, which then goes back to plugins_manager
etc.
> {noformat}
>   File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
line 17, in <module>
>     from airflow import configuration
>   File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
line 31, in <module>
>     from airflow.models import DAG
>   File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
line 62, in <module>
>     from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
>   File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
line 55, in <module>
>     raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor TestExecutor not supported.
> {noformat}
> While attempting {{from airflow.plugins_manager import executors_modules}} cycles right
back where it came from:
> {noformat}
> >>> from airflow.plugins_manager import executor_modules
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
line 31, in <module>
>     from airflow.models import DAG
>   File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
line 62, in <module>
>     from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
>   File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
line 55, in <module>
>     raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor LSFExecutor not supported.
> {noformat}
> To reproduce:
> # Create a custom executor, e.g. by copying {{local_executor.py}} into {{$AIRFLOW_HOME/plugins/test_executor.py}}
and replacing occurrences of "Local" with "Test" in the code, which will produce {{TestExecutor}}
> # Update {{$AIRFLOW_HOME/airflow.cfg}} by setting in the {{\[core\]}} section: {{executor
= TestExecutor}}
> # Run any airflow command, e.g. "airflow version"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message