From dev-return-8490-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Tue May 14 04:57:30 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 64061180671 for ; Tue, 14 May 2019 06:57:30 +0200 (CEST) Received: (qmail 24602 invoked by uid 500); 14 May 2019 04:57:28 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 24590 invoked by uid 99); 14 May 2019 04:57:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 May 2019 04:57:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 48B84180C6A for ; Tue, 14 May 2019 04:57:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2 X-Spam-Level: ** X-Spam-Status: No, score=2 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=thinknear-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id P3rL9BuFTmB4 for ; Tue, 14 May 2019 04:57:25 +0000 (UTC) Received: from mail-ua1-f49.google.com (mail-ua1-f49.google.com [209.85.222.49]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A4A325FDA5 for ; Tue, 14 May 2019 04:57:24 +0000 (UTC) Received: by mail-ua1-f49.google.com with SMTP id 7so1618828uah.1 for ; Mon, 13 May 2019 21:57:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thinknear-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=x6QXXOC3Sqth/t7HMztn5AKBc7pXfeKd+OfD5nE1JGs=; b=e0ArkKsJruSIsdoy+8H1yyInITlwK9Qfz3PdJwdVfoN7xapk6qnBUledWBvXe888OG pIUgQMT9VfrQeA5z0Apx7Zx87f4m2Z+FQ9daOREkj7lTwlQ45JU8PWzuOeTF8brlCY08 DAXylcDtYAkGlFzF1DOKdvWe4ibvv05QXE+s9tQpzkCPXUx9XUeyKDfD1xe+5vqd48tx /bPMqD8b102CVyvHkplzHqT4uQJQl8LwOGXb9W8r+Az/G1kEhayqcoLGt5xeG6jvft0v 01LOpNjoPVTzfmnvoPe5V+MV1iH29h7f99pFAWS6nWz6kjynhnk/My5MhwhYeJGYTuDR Hg8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=x6QXXOC3Sqth/t7HMztn5AKBc7pXfeKd+OfD5nE1JGs=; b=pWusgM9PaWeyMMVowVGAbH/etW9mybV+Axz1DI3G8b9ieFtX3C6xTJQy5qRa+UDEPr +FxuGJ8ElRnXBJppYTvyM/HdNuH5VfAsWrXlyYiP07RonZkdtEYYcraMGFJ7RR/GTzYT zID1sUp+oSIwZrGYUEpqnv/wJPZc+IxpXIwIfp0B9ceObf1MFhLrqw2NrM75vFFl9QRZ vOu/Lvj0VlIlhLcXlzrpY3n4gw8jd8M9YAV6ThUwUY4zLKPxKncJOts/mLc+ljjo0rG4 NY06vv0jagEJ/gf644ZNObTN04YO2/x1mDIkSB2uqx3f3Mkv75bBhNkdyizAugVRzyTr W8wg== X-Gm-Message-State: APjAAAWaKFm1P3FvvHBDoAJ3/TOung/z6ZyPyT6T5ZDMaltQx1WerQ5F cYwVaFMx5U6CAwfPl+3mJBOquRC0zqQ3stZcqal339Ol/mU= X-Google-Smtp-Source: APXvYqxw/cMpKO9ieZod2RmNDyXxMg+66HyjSqVKEP7y7nwZ5CMEdr6A1v/M7SOZWZxnLhEBSiMHYXmpKF5D7YMc7p0= X-Received: by 2002:ab0:806:: with SMTP id a6mr16235119uaf.10.1557809837947; Mon, 13 May 2019 21:57:17 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: David Klosowski Date: Mon, 13 May 2019 21:57:06 -0700 Message-ID: Subject: Re: Cron schedule with DST-aware timezone To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="00000000000004cff10588d1de31" --00000000000004cff10588d1de31 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The distinction could come from being non-UTC VS UTC. You can schedule a DAG at any time UTC and it would follow the 24-hour intervals but if it is non-UTC it would follow DST changes when relevant (by timezone). You could technically make this behavior configurable and follow the former path as well (not sure the use cases). On Mon, May 13, 2019 at 9:17 PM Jarek Potiuk wrote: > Just to add to that: there are already tests that tests this behaviour: > > https://github.com/PolideaInternal/airflow/blob/master/tests/models/test_= dag.py#L749 > and > indeed - the schedule follows DST changes rather than discards the DST > time. > > I think it is generally a good idea to follow DST (for clarity), but I > might miss some context/cases. Maybe it is indeed better to have always 2= 4 > hour intervals for daily schedules for example rather than 23/25 > sometimes). > > BTW. I fixed one of those tests that was failing recently on Python 3.6 C= I. > Python 3.6 behaved a bit differently than 3.5 at DST change: > https://issues.apache.org/jira/browse/AIRFLOW-4308 > > J. > > > > On Mon, May 13, 2019 at 11:39 PM Maxime Beauchemin < > maximebeauchemin@gmail.com> wrote: > > > It would be great if people can provide failing unit tests as PR with > clear > > expectations stated out as code. It makes it easier for people to get > > consensus on expectations and for anyone to jump in and implement a fix= . > > > > Max > > > > On Mon, May 13, 2019 at 12:48 PM David Klosowski > > wrote: > > > > > Damian is correct. We've observed that exact behavior and noticed th= e > > > timedelta logic is dubiously broken for DST but works for CRON. > > > > > > On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. < > > > damian.shaw.2@credit-suisse.com> wrote: > > > > > > > Hi, > > > > > > > > I=E2=80=99m not part of the Airflow team but I came to the same con= clusion, > > that > > > > the behavior is the opposite of what the documentation specifies. > When > > > > using the cron notation then DST is properly honored, when using > > > timedelta > > > > it is not. I played around with the DAG.following_schedule method t= o > > > > satisfy this was the case. > > > > > > > > I=E2=80=99ve had production instance of airflow based on this that > successfully > > > > respected the March/April DST changes in many timezones. > > > > > > > > Regards > > > > Damian > > > > > > > > From: Jiahao Chen [mailto:jhchen@google.com.INVALID] > > > > Sent: Monday, May 13, 2019 2:08 PM > > > > To: dev@airflow.apache.org > > > > Subject: Cron schedule with DST-aware timezone > > > > > > > > Hi team, > > > > > > > > I have a question about the expected behavior of Airflow scheduler > when > > > > the schedule_interval is a cron expression and the start_date is in= a > > > > timezone with DST. > > > > > > > > Based on the Airflow documentation > > > > https://airflow.apache.org/timezone.html#cron-schedules, the DST > > change > > > > will be ignored if schedule_interval is a cron expresion (e.g. '0 1= 7 > * > > * > > > > *'). And it gives an example that the GMT offset will not change > > > regardless > > > > how DST changes. If I'm understanding it correctly, that means if I > > > upload > > > > a DAG with a schedule_interval of "0 17 * * *" and a start_date of > > > > 2019-03-15 17:00 PST(GMT-8) which is before the DST change on March > 10, > > > the > > > > Airflow scheduler will always start the DAG on 5 pm everyday GMT-8 > even > > > > after the DST change on March 10. > > > > > > > > However, that is not the behavior I've seen with my experimental co= de > > > (see > > > > attachments). It looks like the the Airflow is actually taking the > DST > > > into > > > > account, since the execution time is always 17:00 locally, which is= 1 > > > hour > > > > off on the GMT after the DST change. > > > > > > > > Could you please confirm the behavior of Airflow scheduler in this > use > > > > case? > > > > > > > > Thank you! > > > > Jiahao > > > > > > > > > > > > > > > > > > > > > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > > > > > > > > Please access the attached hyperlink for an important electronic > > > > communications disclaimer: > > > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > > > > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > > > > > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > E: jarek.potiuk@polidea.com > --00000000000004cff10588d1de31--