airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-385) Scheduler logs should have a "latest" directory
Date Thu, 20 Oct 2016 21:01:59 GMT


Dan Davydov commented on AIRFLOW-385:

I agree something like this would be very useful, the problem is these logs contain attempts
to parse files for DAGs, so in some cases a DAG might not even be created due to an error
so you couldn't have dag_id/execution_date/...

There is probably a middle-ground, maybe where we have a duplicate log folder for DAGs that
have been successfully parsed with symlinks pointing back to the original log directory, but
this idea scares me (symlinks, and duplication). Another idea would be a CLI tool like airflow
view_scheduler_logs dag_id task_id execution_date.

In any case I feel this issue is orthogonal to the "latest" symlink (as this symlink would
be useful even in the dag_id/task_id/execution_date model), but definitely feel free to open
another ticket for it.

> Scheduler logs should have a "latest" directory
> -----------------------------------------------
>                 Key: AIRFLOW-385
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: logging
>            Reporter: Dan Davydov
>            Assignee: Vijay Bhat
>            Priority: Minor
>              Labels: beginner, logging
> Right now the scheduler logs for each parsed file look like /tmp/airflow/scheduler/logs/2016-07-17/some-dag/...
> The problem is this makes it hard to find the latest logs which is the most frequent
use case.
> Ideally we would create and have the scheduler keep up to date a symlink to the latest
date /tmp/airflow/scheduler/logs/latest which would point to e.g. /tmp/airflow/scheduler/logs/2016-07-17
> We might also want to consider changing the structure to dag/date instead of date/dag
too, but that can be done as a separate task if desired.

This message was sent by Atlassian JIRA

View raw message