Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CABYbY7cKLupcrFTeK5E-HT4eJt33YKw9+bJ5RfbdbTBUC1=ExQ@mail.gmail.com>
References: <CAB=riaBq7oC7XQVio6BaHv7bbe+_5hvJ+bwvn-=9iO0dtwptOA@mail.gmail.com>
 <CAB=riaDFErWay=P-a32rCVYah+++kKPRGcxd48Up0Mxhof2ipA@mail.gmail.com>
 <CAB=riaDCkBfQF7Qh36XuQFfhWW6Z4ssgFBnBu51GmR+1najbJg@mail.gmail.com>
 <CAB=riaCm-hjF2OvB8=VeP+O4r4eNU4RCiZOKWxyO3kuyUyJOgA@mail.gmail.com>
 <CAB=riaC8KgqQjN0PKvYWH009gZcEmH9kyBmZSjfjr7R6iR9HyQ@mail.gmail.com>
 <CAB=riaBCEOuK89mcXmtU8U4T1pXUcDHV8waTr=V26r4q3W-NAw@mail.gmail.com>
 <CAB=riaBbQLKbVn=ZK5wZPvjdYs0yBLggiFfqAXLVrfWFMwyFxQ@mail.gmail.com>
 <CAB=riaC13mm7irfJ5Fg068VvJ_1PruBfB7Yv6Zbv+ncnBP-AQg@mail.gmail.com>
 <CAB=riaCWsvzSB8TzrE_GputEtftW5csLA=gw42LxcoDLsOJucw@mail.gmail.com>
 <CAB=riaA8bV9u=pEoes7-XOWQ1WTiJm=q0wnODWZbb1o1T5EEvQ@mail.gmail.com>
 <CAB=riaBBnMwhf5D0pxky-QoqZemkbtvbR=Wk-GsSXn52RsynxQ@mail.gmail.com>
 <CAB=riaBydwJyN8S5QrfSQDS_sFGOk+278b0qE2g=MHeP-jDnWw@mail.gmail.com>
 <CAFGbz6U+A0B58hF9ahb_bcSv_40t=jfQ5SOqR18SK021pEvyEA@mail.gmail.com>
 <CAHEEp7WNxE15dA5JD3aTtAht7yAwAWXUuFSPca+JAHumUJWSow@mail.gmail.com> <CABYbY7cKLupcrFTeK5E-HT4eJt33YKw9+bJ5RfbdbTBUC1=ExQ@mail.gmail.com>
From: David Capwell <dcapwell@gmail.com>
Date: Tue, 29 Aug 2017 11:02:33 -0700
Message-ID: <CAB=riaDeo0x9hNFEqwQj_6UYi_RwHxU1sYqnTz2vEcbDB9E+1g@mail.gmail.com>
Subject: Re: As history grows UI gets slower
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary="94eb2c1954ea4484de0557e837cf"
archived-at: Tue, 29 Aug 2017 18:02:42 -0000

--94eb2c1954ea4484de0557e837cf
Content-Type: text/plain; charset="UTF-8"

Thanks, will take a look at this project

On Aug 29, 2017 10:35 AM, "Chris Riccomini" <criccomini@apache.org> wrote:

> Might have a look at this, too:
>
> https://github.com/teamclairvoyant/airflow-maintenance-dags
>
> I haven't used it, but it seems to have a DB cleaning script.
>
> On Mon, Aug 28, 2017 at 3:43 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Just make sure to archive based on start_date and not execution_date to
> > allow backfills.
> >
> > Max
> >
> > On Mon, Aug 28, 2017 at 3:20 PM, Alex Guziel <alex.guziel@airbnb.com.
> > invalid
> > > wrote:
> >
> > > Here at Airbnb we delete old "completed" task instances.
> > >
> > > On Mon, Aug 28, 2017 at 3:01 PM, David Capwell <dcapwell@gmail.com>
> > wrote:
> > >
> > > > We are on 1.8.0 and have a monitor DAG that monitors the health of
> > > Airflow
> > > > and Celery every minute.  This has been running for awhile now and at
> > 26k
> > > > dag runs. We see that the UI for this DAG is multiple seconds slower
> > (6-7
> > > > second) than any other DAG.
> > > >
> > > > My question is, what do people do about managing history as it grows
> > over
> > > > time? Do people delete history after N or so days?
> > > >
> > > > Thanks for your time reading this email
> > > >
> > >
> >
>

--94eb2c1954ea4484de0557e837cf--