airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-245) Access to task instance from custom Executor
Date Tue, 21 Jun 2016 13:44:57 GMT


Jeremiah Lowin commented on AIRFLOW-245:

The Scheduler is an object that decides what tasks to run; the Executor is an object that
decides where to run those tasks. So the Scheduler hands a TaskInstance to the Executor and
says "I want to run this task". That's why the Executor has access to the TaskInstance when
tasks are being queued. The Executor then extracts the TI's "command" (for example: airflow
run my_task) and its sole job is to decide how to run that command. execute_async() is where
that decision is made. The important thing is that the thing being run is the string command,
not the TI object, and the signature of execute_async should reflect that. 

The larger question here is why would an operator ever influence the executor? An Operator
is a series of instructions that are designed to be run in a self-contained way, no matter
what the executor is. An executor is a way of kicking off an Operator. There should be no
dependency between them. To be explicit: an Operator must be able to run under any Executor.

However, I am not familiar with Mesos so unfortunately I don't know exactly what you're trying
to do -- but have you looked at the existing MesosExecutor in airflow.contrib.executors? Is
it insufficient for your objectives? If your operator requires special handling, then it sounds
like maybe you want a MesosOperator that runs your docker command?

> Access to task instance from custom Executor
> --------------------------------------------
>                 Key: AIRFLOW-245
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executor
>            Reporter: Alexandr Nikitin
> I'm writing a custom executor that executes tasks on mesos and I want to have access
to task instances from it. So that I can reuse all existing operators e.g. DockerOperator
and access its fields like image, command, volumes and transform them to mesos.
> This can be done by changing `def execute_async(self, key, command, queue=None):` in

This message was sent by Atlassian JIRA

View raw message