airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laura Lorenz <llor...@industrydive.com>
Subject Re: Airflow pause-all
Date Fri, 01 Jul 2016 15:10:57 GMT
Before a given DAG has a history; if the metadata database has it listed as
unpaused, the scheduler will honor that.

On Fri, Jul 1, 2016 at 11:05 AM, Tyrone Hinderson <thinderson@reonomy.com>
wrote:

> I found dags_are_paused_at_creation, but what does "at creation" really
> mean? Every time the scheduler starts, or before a given DAG has any
> history?
>
> On Fri, Jul 1, 2016 at 10:39 AM Tyrone Hinderson <thinderson@reonomy.com>
> wrote:
>
> > Interesting, is that config variable in the "core" section?
> >
> > And environment variable config is still a thing in 1.7.1.2, if you're
> > interested.
> >
> > On Thu, Jun 30, 2016 at 5:08 PM Laura Lorenz <llorenz@industrydive.com>
> > wrote:
> >
> >> Possibly useful to you is the variable dags_are_paused_at_creation =
> True
> >> you
> >> can set in airflow.cfg or in environment variables to cause the metadata
> >> database to consider all DAGs paused when they are first started up.
> >>
> >> I do think I saw something fly by on this dev list that environment
> >> variables for config got written out in the newest version. Not sure if
> >> it's true as I haven't tried to upgrade from 1.7.0 yet... partially for
> >> fear of tearing out our dependence on being able to control other
> airflow
> >> config with environment variables :)
> >>
> >> On Thu, Jun 30, 2016 at 4:59 PM, Tyrone Hinderson <
> thinderson@reonomy.com
> >> >
> >> wrote:
> >>
> >> > Hey there Chris,
> >> >
> >> > I agree that's a solid use case--mine is more development-oriented.
> >> Rather
> >> > than have all DAGs attempt to run upon scheduler start, a developer
> >> > probably just wants to observe how one DAG behaves. To that end, a
> >> > scheduler startup script can simply pause all DAGs before starting the
> >> dev
> >> > instance of the scheduler; then the developer can selectively unpause
> >> that
> >> > DAG via the UI or CLI (assuming those two controls are related).
> >> >
> >> > On Thu, Jun 30, 2016 at 4:06 PM Chris Riccomini <
> criccomini@apache.org>
> >> > wrote:
> >> >
> >> > > Hey Tyrone,
> >> > >
> >> > > This would be a useful feature for us as well. Right now, we just
> turn
> >> > the
> >> > > scheduler off, which is kind of annoying.
> >> > >
> >> > > Use case is when things go really haywire (e.g. 100s of failures per
> >> > > minute), we just want everything to stop.
> >> > >
> >> > > Cheers,
> >> > > Chris
> >> > >
> >> > > On Thu, Jun 30, 2016 at 1:02 PM, Tyrone Hinderson <
> >> > thinderson@reonomy.com>
> >> > > wrote:
> >> > >
> >> > > > Hi, is there a way to "pause" all DAGs via the cli or otherwise?
> >> > > >
> >> > > > In the same vein, does "airflow pause" flip the same state which
> is
> >> > > flipped
> >> > > > by the on/off button next to each DAG in the UI?
> >> > > >
> >> > >
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message