airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacopo Sabbatini (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AIRFLOW-3156) AWS Batch executor
Date Tue, 09 Oct 2018 08:24:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642949#comment-16642949
] 

Jacopo Sabbatini edited comment on AIRFLOW-3156 at 10/9/18 8:24 AM:
--------------------------------------------------------------------

Well in a sense it's like the use case for the Mesos and Kubernetes executor.

Correct me if wrong but when using the AWS Batch operator all the task does is continuously
polling for the task status until done. This seems a very inefficient use of a worker. Furthermore
if I wanted to submit many tasks using the Batch operator I suspect I would start seeing the
polling API requests being throttled by AWS.

Having a Batch executor it would mean that the polling of the job status can be batched using
the DescribeJobs Batch API this would make having many tasks more efficient. Using Batch as
executor would also simplify deployments since it remove the needs for workers that are now
managed by AWS Batch.


was (Author: dedalusj):
Well in a sense it's like the use case for the Mesos and Kubernetes executor.

Correct me if wrong but when using the AWS Batch operator all the task does is continuously
polling for the task status until done. This seems a very inefficient use of a worker. Furthermore
if I wanted to submit many tasks using the Batch operator I suspect I would start seeing the
polling API requests being throttled by AWS.

Having a Batch executor it would mean that the polling of the job status can be batched using
the DescribeJobs Batch API this would make having many tasks more efficient.

> AWS Batch executor
> ------------------
>
>                 Key: AIRFLOW-3156
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3156
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: executor
>            Reporter: Jacopo Sabbatini
>            Priority: Major
>              Labels: aws
>
> Add support for AWS Batch as executor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message