airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [DISCUSS] Utility DAGs
Date Mon, 01 Apr 2019 09:54:15 GMT
> For example, say we want to support retention in some Airflow tables such
> as task_instance, dag_run and log, it seems reasonable to me to create a
> DAG to periodically clean up the tables

I guess you mean something like https://github.com/teamclairvoyant/airflow-maintenance-dags/tree/master/db-cleanup
but just shipped with Airflow?

The reason I don't think this should be best as a DAG in Airflow is that we can do it better/cleaner
if it is core to Airflow:

- We don't need to speculatively run a DAG that does nothing
- We don't need to "waste" an executor slot
- It could automatically be done before/after running another task in the dag
- We don't create extra task instance rows/logs that we than have to clean up too.

That is my thinking of why I don't think this sort of built-in functionality should be a DAG
if it is shipped _with_ Airflow.

-ash


> On 31 Mar 2019, at 20:42, Chao-Han Tsai <milton0825@gmail.com> wrote:
> 
> Thanks Ash and Kevin for the feedback.
> 
> I think there are some utilities that can be solved easily with a DAG
> without introducing more logic to complicate the scheduler code. Also,
> these utilities may run periodically and can be abstracted out with a DAG.
> For example, say we want to support retention in some Airflow tables such
> as task_instance, dag_run and log, it seems reasonable to me to create a
> DAG to periodically clean up the tables.
> 
> Would like to learn more about the concerns about introducing these utility
> DAGs.
> 
> On Sun, Mar 31, 2019 at 1:17 AM Kevin Yang <yrqls21@gmail.com> wrote:
> 
>> Agree on having core airflow related stuff built into airflow( like
>> schedule delay instrumentation) and leave the others to cluster maintainer
>> to set up( like log retention). How people handle log retention might be
>> quite different depends on the logging backend. E.g. we use ElasticSearch
>> and we don't even manage the log retention ourselves. Same for stuff like
>> metrics/ alert submitting.
>> 
>> Just my $0.02
>> 
>> Cheers,
>> Kevin Y
>> 
>> On Sun, Mar 31, 2019 at 12:48 AM Ash Berlin-Taylor <ash@apache.org> wrote:
>> 
>>> Do these need to me dags of they are built in to Airflow, or could/should
>>> they be just handled internally by the scheduler?
>>> 
>>> -a
>>> 
>>> On 31 March 2019 03:57:08 BST, Chao-Han Tsai <milton0825@gmail.com>
>> wrote:
>>>> Hi all,
>>>> 
>>>> I have been thinking about adding some DAGs that are for the purpose of
>>>> AIrflow cluster operation, DAG schedule delay instrumentation and log
>>>> retention for instance. Currently we have example_dags, should we add
>>>> another directory utility_dags in the repo? We can have a flag in
>>>> airflow.cfg to let user decide whether to load the utility_dags (just
>>>> like
>>>> what we did for example_dags).
>>>> 
>>>> What do you think?
>>>> 
>>>> --
>>>> Chao-Han Tsai
>>> 
>> 
> 
> 
> -- 
> 
> Chao-Han Tsai


Mime
View raw message