airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2068) Mesos Executor should allow specifying optional Docker image before running command
Date Mon, 23 Apr 2018 20:24:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448806#comment-16448806
] 

ASF subversion and git services commented on AIRFLOW-2068:
----------------------------------------------------------

Commit d96534c268a13f8e6a0e1e288821a436322926c4 in incubator-airflow's branch refs/heads/v1-10-test
from [~amangal]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d96534c ]

[AIRFLOW-2068] MesosExecutor allows optional Docker image

In its current form, MesosExecutor schedules tasks
on mesos slaves which
just contain airflow commands assuming that the
mesos slaves already
have airflow installed and configured on them.
This assumption goes
against the Mesos philosophy of having a
heterogeneous cluster.

Since Mesos provides an option to pull a Docker
image before actually
running the actual task/command so this
improvement changes the
mesos_executor.py to specify an optional docker
image containing
airflow which can be pulled on slaves before
running the actual
airflow command. This also opens the door for an
optimization of
resources in a future PR, by allowing the
specification of CPU and
memory needed for each airflow task.

Closes #3008 from agrajm/AIRFLOW-2068

(cherry picked from commit 1f86299cf998729ea93c666481390451c9724ebc)
Signed-off-by: Bolke de Bruin <bolke@xs4all.nl>


> Mesos Executor should allow specifying optional Docker image before running command
> -----------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-2068
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2068
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executor
>    Affects Versions: Airflow 1.8
>            Reporter: Agraj Mangal
>            Assignee: Agraj Mangal
>            Priority: Major
>             Fix For: 1.10.0
>
>
> In its current form, MesosExecutor schedules tasks on mesos slaves which just contain
airflow commands assuming that the mesos slaves already have airflow installed and configured
on them. This assumption goes against the Mesos philosophy of having a heterogeneous cluster.
> Since mesos provides an option to pull a docker image before actually running the actual
task/command so this improvement changes the mesos_executor.py to specify an optional docker
image containing airflow which can be pulled on slaves before running the actual airflow
command. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message