airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Klosowski <dav...@thinknear.com>
Subject Re: Cron schedule with DST-aware timezone
Date Tue, 14 May 2019 04:57:06 GMT
The distinction could come from being non-UTC VS UTC.  You can schedule a
DAG at any time UTC and it would follow the 24-hour intervals but if it is
non-UTC it would follow DST changes when relevant (by timezone).  You could
technically make this behavior configurable and follow the former path as
well (not sure the use cases).

On Mon, May 13, 2019 at 9:17 PM Jarek Potiuk <Jarek.Potiuk@polidea.com>
wrote:

> Just to add to that: there are already tests that tests this behaviour:
>
> https://github.com/PolideaInternal/airflow/blob/master/tests/models/test_dag.py#L749
> and
> indeed - the schedule follows DST changes rather than discards the DST
> time.
>
> I think it is generally a good idea to follow DST (for clarity), but I
> might miss some context/cases. Maybe it is indeed better to have always 24
> hour intervals for daily schedules for example rather than 23/25
> sometimes).
>
> BTW. I fixed one of those tests that was failing recently on Python 3.6 CI.
> Python 3.6 behaved a bit differently than 3.5 at DST change:
> https://issues.apache.org/jira/browse/AIRFLOW-4308
>
> J.
>
>
>
> On Mon, May 13, 2019 at 11:39 PM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > It would be great if people can provide failing unit tests as PR with
> clear
> > expectations stated out as code. It makes it easier for people to get
> > consensus on expectations and for anyone to jump in and implement a fix.
> >
> > Max
> >
> > On Mon, May 13, 2019 at 12:48 PM David Klosowski <davidk@thinknear.com>
> > wrote:
> >
> > > Damian is correct.  We've observed that exact behavior and noticed the
> > > timedelta logic is dubiously broken for DST but works for CRON.
> > >
> > > On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. <
> > > damian.shaw.2@credit-suisse.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I’m not part of the Airflow team but I came to the same conclusion,
> > that
> > > > the behavior is the opposite of what the documentation specifies.
> When
> > > > using the cron notation then DST is properly honored, when using
> > > timedelta
> > > > it is not. I played around with the DAG.following_schedule method to
> > > > satisfy this was the case.
> > > >
> > > > I’ve had production instance of airflow based on this that
> successfully
> > > > respected the March/April DST changes in many timezones.
> > > >
> > > > Regards
> > > > Damian
> > > >
> > > > From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
> > > > Sent: Monday, May 13, 2019 2:08 PM
> > > > To: dev@airflow.apache.org
> > > > Subject: Cron schedule with DST-aware timezone
> > > >
> > > > Hi team,
> > > >
> > > > I have a question about the expected behavior of Airflow scheduler
> when
> > > > the schedule_interval is a cron expression and the start_date is in a
> > > > timezone with DST.
> > > >
> > > > Based on the Airflow documentation
> > > > https://airflow.apache.org/timezone.html#cron-schedules, the DST
> > change
> > > > will be ignored if schedule_interval is a cron expresion (e.g. '0 17
> *
> > *
> > > > *'). And it gives an example that the GMT offset will not change
> > > regardless
> > > > how DST changes. If I'm understanding it correctly, that means if I
> > > upload
> > > > a DAG with a schedule_interval of "0 17 * * *" and a start_date of
> > > > 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March
> 10,
> > > the
> > > > Airflow scheduler will always start the DAG on 5 pm everyday GMT-8
> even
> > > > after the DST change on March 10.
> > > >
> > > > However, that is not the behavior I've seen with my experimental code
> > > (see
> > > > attachments). It looks like the the Airflow is actually taking the
> DST
> > > into
> > > > account, since the execution time is always 17:00 locally, which is 1
> > > hour
> > > > off on the GMT after the DST change.
> > > >
> > > > Could you please confirm the behavior of Airflow scheduler in this
> use
> > > > case?
> > > >
> > > > Thank you!
> > > > Jiahao
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> ===============================================================================
> > > >
> > > > Please access the attached hyperlink for an important electronic
> > > > communications disclaimer:
> > > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> > > >
> > >
> >
> ===============================================================================
> > > >
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> E: jarek.potiuk@polidea.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message