airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fang-Pen Lin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-2325) Task logging with AWS Cloud watch
Date Mon, 16 Apr 2018 02:32:00 GMT
Fang-Pen Lin created AIRFLOW-2325:
-------------------------------------

             Summary: Task logging with AWS Cloud watch
                 Key: AIRFLOW-2325
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2325
             Project: Apache Airflow
          Issue Type: New Feature
          Components: logging
            Reporter: Fang-Pen Lin


In many cases, it's ideal to use remote logging while running Airflow in production, as the
worker could be easily scale down or scale up. Or the worker is running in containers, where
the local storage is not meant to be there forever. In that case, the S3 task logging handler
could be used

[https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/s3_task_handler.py]

However, it comes with drawback. S3 logging handler only uploads the log when the task completed
or failed. For long running tasks, it's hard to know what's going on with the process until
it finishes.

To make more real-time logging, I built a logging handler based on AWS CloudWatch. It uses
a third party python package `watchtower`

 

[https://github.com/kislyuk/watchtower/tree/master/watchtower]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message