airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-5027) Generalized CloudWatch log grabbing for ECS and SageMaker operators
Date Thu, 22 Aug 2019 14:45:01 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913393#comment-16913393
] 

ASF subversion and git services commented on AIRFLOW-5027:
----------------------------------------------------------

Commit 786db8b0ee181135017c976c5ccb87de9f2e9025 in airflow's branch refs/heads/master from
Steven Reitsma
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=786db8b ]

[AIRFLOW-5027] Grab CloudWatch logs after ECS task has finished (#5645)



> Generalized CloudWatch log grabbing for ECS and SageMaker operators
> -------------------------------------------------------------------
>
>                 Key: AIRFLOW-5027
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5027
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: aws, contrib, logging
>    Affects Versions: 1.10.3, 1.10.4
>            Reporter: Steven Reitsma
>            Assignee: Steven Reitsma
>            Priority: Major
>
> It's very useful to have the CloudWatch task logs of an ECS/SageMaker job available within
Airflow. Currently, this is not possible for ECS tasks.
> The SageMaker operator starts a job on AWS SageMaker. It already supports grabbing the
CloudWatch logs of the (finished) job to the Airflow instance.
> The ECS operator is very similar in that it starts a job on AWS ECS. It doesn't, however,
support grabbing CloudWatch logs at the end of a job.
> I've separated the existing log grabbing of the SageMaker operator to a hook and both
the SageMaker and ECS operators now call that hook at the end of a job.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message