airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <lance.nors...@gmail.com>
Subject Re: "intra-day" backfill mistake
Date Fri, 03 Jun 2016 06:56:45 GMT
In general, since a DAG declares its own periodicity, why should any
reference to the DAG need to also be precise?
Shouldn't an incomplete timestamp refer to "the next time DAG happens"?


On Tue, May 31, 2016 at 9:05 AM, Chris Riccomini <criccomini@apache.org>
wrote:

> I agree. The date format handling in the CLI is not intuitive. IMO, it
> should force full-resolution YYYYMMDDHHMMSS.micro.
>
> On Fri, May 27, 2016 at 7:28 PM, Bence Nagy <bence@underyx.me> wrote:
>
> > I can think of weird legit reasons why one would force running the dag on
> > an intra-day start date, but a warning message requiring confirmation
> > (perhaps also offering to autocorrect the date to the nearest schedule
> > matching one) would be awesome to have.
> >
> > On Sat, May 28, 2016, 3:22 AM Lance Norskog <lance.norskog@gmail.com>
> > wrote:
> >
> > > I just made the same mistake, twice.
> > >
> > > I have a Dag with this schedule:
> > >
> > > dag = DAG(
> > >     dag_id='xyz', default_args=args,
> > >     schedule_interval='0 2 * * *',
> > >     start_date=datetime(2016, 5, 16,2),
> > >     max_active_runs=1
> > >     )
> > >
> > >
> > > It runs daily, at 2AM UTC. I ran this command:
> > > airflow backfill -s 2016-05-24 xyz
> > >
> > > So, of course, it created a new run at midnight UTC instead of running
> > the
> > > DAG at 2am.
> > >
> > > Should 'airflow backfill' and similar respect the periodicity of the
> DAG?
> > > Could these command give an error and require a flag to force running
> > > outside the DAG's periodicity?
> > >
> > > --
> > > Lance Norskog
> > > lance.norskog@gmail.com
> > > Redwood City, CA
> > >
> >
>



-- 
Lance Norskog
lance.norskog@gmail.com
Redwood City, CA

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message