airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Cron schedule with DST-aware timezone
Date Tue, 14 May 2019 04:17:10 GMT
Just to add to that: there are already tests that tests this behaviour:
https://github.com/PolideaInternal/airflow/blob/master/tests/models/test_dag.py#L749
and
indeed - the schedule follows DST changes rather than discards the DST time.

I think it is generally a good idea to follow DST (for clarity), but I
might miss some context/cases. Maybe it is indeed better to have always 24
hour intervals for daily schedules for example rather than 23/25 sometimes).

BTW. I fixed one of those tests that was failing recently on Python 3.6 CI.
Python 3.6 behaved a bit differently than 3.5 at DST change:
https://issues.apache.org/jira/browse/AIRFLOW-4308

J.



On Mon, May 13, 2019 at 11:39 PM Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> It would be great if people can provide failing unit tests as PR with clear
> expectations stated out as code. It makes it easier for people to get
> consensus on expectations and for anyone to jump in and implement a fix.
>
> Max
>
> On Mon, May 13, 2019 at 12:48 PM David Klosowski <davidk@thinknear.com>
> wrote:
>
> > Damian is correct.  We've observed that exact behavior and noticed the
> > timedelta logic is dubiously broken for DST but works for CRON.
> >
> > On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. <
> > damian.shaw.2@credit-suisse.com> wrote:
> >
> > > Hi,
> > >
> > > I’m not part of the Airflow team but I came to the same conclusion,
> that
> > > the behavior is the opposite of what the documentation specifies. When
> > > using the cron notation then DST is properly honored, when using
> > timedelta
> > > it is not. I played around with the DAG.following_schedule method to
> > > satisfy this was the case.
> > >
> > > I’ve had production instance of airflow based on this that successfully
> > > respected the March/April DST changes in many timezones.
> > >
> > > Regards
> > > Damian
> > >
> > > From: Jiahao Chen [mailto:jhchen@google.com.INVALID]
> > > Sent: Monday, May 13, 2019 2:08 PM
> > > To: dev@airflow.apache.org
> > > Subject: Cron schedule with DST-aware timezone
> > >
> > > Hi team,
> > >
> > > I have a question about the expected behavior of Airflow scheduler when
> > > the schedule_interval is a cron expression and the start_date is in a
> > > timezone with DST.
> > >
> > > Based on the Airflow documentation
> > > https://airflow.apache.org/timezone.html#cron-schedules, the DST
> change
> > > will be ignored if schedule_interval is a cron expresion (e.g. '0 17 *
> *
> > > *'). And it gives an example that the GMT offset will not change
> > regardless
> > > how DST changes. If I'm understanding it correctly, that means if I
> > upload
> > > a DAG with a schedule_interval of "0 17 * * *" and a start_date of
> > > 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March 10,
> > the
> > > Airflow scheduler will always start the DAG on 5 pm everyday GMT-8 even
> > > after the DST change on March 10.
> > >
> > > However, that is not the behavior I've seen with my experimental code
> > (see
> > > attachments). It looks like the the Airflow is actually taking the DST
> > into
> > > account, since the execution time is always 17:00 locally, which is 1
> > hour
> > > off on the GMT after the DST change.
> > >
> > > Could you please confirm the behavior of Airflow scheduler in this use
> > > case?
> > >
> > > Thank you!
> > > Jiahao
> > >
> > >
> > >
> > >
> > >
> >
> ===============================================================================
> > >
> > > Please access the attached hyperlink for an important electronic
> > > communications disclaimer:
> > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> > >
> >
> ===============================================================================
> > >
> > >
> >
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
E: jarek.potiuk@polidea.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message