airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mad2271@columbia.edu <mad2...@columbia.edu>
Subject How to consolidate log files?
Date Mon, 30 Apr 2018 19:16:58 GMT
Hi Mailing List,

Is there a way to consolidate airflow task logs by stdout and stderr? Currently, the structure
is something like /logs/taskname/task_run_date/log1.txt which is the log for a particular
taskname at a particular run date.  What I would like is two large log files for all tasks,
something like /logs/errors.txt and /logs/outputs.txt  Which would contain all the stderr
and stdout messages for all runs of all tasks regardless of run date. I essentially want two
very large log files.  For example, if I have task A and task B, instead of having two directories
and then subdirectories for A and B, I would just like two files one with errors from A and
B and one for outputs from A and B. Does airflow provide this information?

Thanks!

Mime
View raw message