airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ash Berlin-Taylor (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (AIRFLOW-5027) Generalized CloudWatch log grabbing for ECS and SageMaker operators
Date Thu, 22 Aug 2019 15:08:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ash Berlin-Taylor resolved AIRFLOW-5027.
----------------------------------------
    Fix Version/s: 1.10.5
       Resolution: Done

> Generalized CloudWatch log grabbing for ECS and SageMaker operators
> -------------------------------------------------------------------
>
>                 Key: AIRFLOW-5027
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5027
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: aws, contrib, logging
>    Affects Versions: 1.10.3, 1.10.4
>            Reporter: Steven Reitsma
>            Assignee: Steven Reitsma
>            Priority: Major
>             Fix For: 1.10.5
>
>
> It's very useful to have the CloudWatch task logs of an ECS/SageMaker job available within
Airflow. Currently, this is not possible for ECS tasks.
> The SageMaker operator starts a job on AWS SageMaker. It already supports grabbing the
CloudWatch logs of the (finished) job to the Airflow instance.
> The ECS operator is very similar in that it starts a job on AWS ECS. It doesn't, however,
support grabbing CloudWatch logs at the end of a job.
> I've separated the existing log grabbing of the SageMaker operator to a hook and both
the SageMaker and ECS operators now call that hook at the end of a job.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message