airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Darren Weber (Jira)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-6245) Allow custom waiters for AWS batch jobs
Date Mon, 23 Dec 2019 19:33:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Darren Weber updated AIRFLOW-6245:
----------------------------------
    Fix Version/s: 1.10.8
                   2.0.0

> Allow custom waiters for AWS batch jobs
> ---------------------------------------
>
>                 Key: AIRFLOW-6245
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6245
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: aws
>    Affects Versions: 1.10.6
>            Reporter: Darren Weber
>            Assignee: Darren Weber
>            Priority: Minor
>              Labels: AWS, aws-batch
>             Fix For: 2.0.0, 1.10.8
>
>
> The botocore waiter for AWS batch jobs has not been merged and released for several years,
i.e.
> - [https://github.com/boto/botocore/pull/1307]
> While working on this Airflow issue, I've also pushed up a PR on botocore to use a default
waiter with exponential backoff and add an option to use a custom function for the delays
between polling status, see
> - [https://github.com/boto/botocore/issues/1915]
> - [https://github.com/boto/botocore/pull/1921]
>  
> For Airflow, adopt something from botocore PR-1307 as an example to create a default
batch job waiter in Airflow.  As a guide to creating a custom waiter, see
> - [https://www.2ndwatch.com/blog/use-waiters-boto3-write/]
> - [https://boto3.amazonaws.com/v1/documentation/api/latest/guide/clients.html#waiters]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message