airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: As history grows UI gets slower
Date Mon, 28 Aug 2017 22:43:20 GMT
Just make sure to archive based on start_date and not execution_date to
allow backfills.

Max

On Mon, Aug 28, 2017 at 3:20 PM, Alex Guziel <alex.guziel@airbnb.com.invalid
> wrote:

> Here at Airbnb we delete old "completed" task instances.
>
> On Mon, Aug 28, 2017 at 3:01 PM, David Capwell <dcapwell@gmail.com> wrote:
>
> > We are on 1.8.0 and have a monitor DAG that monitors the health of
> Airflow
> > and Celery every minute.  This has been running for awhile now and at 26k
> > dag runs. We see that the UI for this DAG is multiple seconds slower (6-7
> > second) than any other DAG.
> >
> > My question is, what do people do about managing history as it grows over
> > time? Do people delete history after N or so days?
> >
> > Thanks for your time reading this email
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message