airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-935) Impossible to use plugin executors
Date Sat, 13 May 2017 10:58:04 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009265#comment-16009265
] 

ASF subversion and git services commented on AIRFLOW-935:
---------------------------------------------------------

Commit 08a784ede6ffe1a2389255e1c6d597e9b2131080 in incubator-airflow's branch refs/heads/master
from [~stverhae]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=08a784e ]

[AIRFLOW-860][AIRFLOW-935] Fix plugin executor import cycle and executor selection

When a plugin is made with a custom Operator and
executor, an import cycle occurs when the executor
is chosen in airflow.cfg because the
executors/__init__.py starts loading plugins too
early.
changed DEFAULT_EXECUTOR use to a function call
which returns the default executor. This lazy
approach fixes the import cycle.

revision eb5982d (included in 1.8) breaks plugin
executors altogether. It makes a new module for
every plugin, so import statements need to be
adapted, but the executor selection is left
unchanged, so it ends up assigning the plugin
module as an executor.
fixed executor selection to work with the new
plugin modules system introduced in 1.8. in
Airflow.cfg a executor can now be specified as
{plugin_name}.{executor_name}

Fixes:
 -
https://issues.apache.org/jira/browse/AIRFLOW-860
 -
https://issues.apache.org/jira/browse/AIRFLOW-935

Closes #2120 from stverhae/master


> Impossible to use plugin executors
> ----------------------------------
>
>                 Key: AIRFLOW-935
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-935
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: executor, plugins
>    Affects Versions: Airflow 1.8, 1.9.0
>            Reporter: Stijn Verhaegen
>
> revision eb5982d (included in 1.8) breaks plugin executors altogether. It makes a new
module for every plugin, so import statements need to be adapted, but the executor selection
is left unchanged, so it ends up assigning the plugin module as an executor.
> To reproduce: take any plugin which defines a custom executor and try to get it loaded
by setting `executor` in the airflow.cfg



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message