airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-4843) Allow orchestration of tasks with Docker Swarm aka `SwarmOperator`
Date Wed, 18 Dec 2019 16:48:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999326#comment-16999326
] 

ASF subversion and git services commented on AIRFLOW-4843:
----------------------------------------------------------

Commit 29cd0babf42a200671bbcf6b5df627735417c2f9 in airflow's branch refs/heads/v1-10-test
from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=29cd0ba ]

fixup! [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489)


> Allow orchestration of tasks with Docker Swarm aka `SwarmOperator`
> ------------------------------------------------------------------
>
>                 Key: AIRFLOW-4843
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4843
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: operators
>    Affects Versions: 1.10.3
>            Reporter: Akshesh Doshi
>            Assignee: Akshesh Doshi
>            Priority: Major
>              Labels: Docker, docker, orchestration, swarm
>             Fix For: 1.10.7
>
>
> Currently, Airflow supports spawning Docker containers for running tasks via the {color:#707070}_DockerOperator_{color}
but these containers are run on the same node as the scheduler.
> It would be helpful for our use-case to be able to spawn these tasks wherever resources
are available in our Docker Swarm cluster.
>  
> This can be achieved by creating a Docker swarm service, waiting for its run and removing
it after it has completed execution.
> This approach has been suggested/discussed at various places (and implemented in Golang
for Swarm-cronjob):
> [https://blog.alexellis.io/containers-on-swarm/]
> [https://forums.docker.com/t/running-one-off-commands-in-swarm-containers/42436/3]
> [https://gist.github.com/alexellis/e11321b8fbfc595c208ea3e74bf5e54b]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message