airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-245) Access to task instance from custom Executor
Date Mon, 20 Jun 2016 16:53:05 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339888#comment-15339888
] 

Jeremiah Lowin commented on AIRFLOW-245:
----------------------------------------

I also posted this on PR 1596 but perhaps it's more relevant here:

Something important to keep in mind -- the current version of execute_async() just requires
the task key (a bunch of strings) and the command that runs the task (another string, something
like airflow run task). That's because we make no assumptions about how the execution is implemented,
or on what machine/environment, or even when it will be run. Passing a task object to the
execute_async() method is contrary to that assumption. In particular, there is no guarantee
that task objects are serializable.

The key should give enough information to load the task inside the executor by querying the
database for the matching taskinstance -- I think that would be the far preferable way to
implement the desired behavior in a custom executor.

> Access to task instance from custom Executor
> --------------------------------------------
>
>                 Key: AIRFLOW-245
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-245
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executor
>            Reporter: Alexandr Nikitin
>
> I'm writing a custom executor that executes tasks on mesos and I want to have access
to task instances from it. So that I can reuse all existing operators e.g. DockerOperator
and access its fields like image, command, volumes and transform them to mesos.
> This can be done by changing `def execute_async(self, key, command, queue=None):` in
`BaseExecutor`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message