From dev-return-4898-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Wed Apr 18 20:26:44 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6D0FF18064E for ; Wed, 18 Apr 2018 20:26:44 +0200 (CEST) Received: (qmail 91533 invoked by uid 500); 18 Apr 2018 18:26:38 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 91521 invoked by uid 99); 18 Apr 2018 18:26:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Apr 2018 18:26:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 56BE4C0188 for ; Wed, 18 Apr 2018 18:26:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.148 X-Spam-Level: ** X-Spam-Status: No, score=2.148 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id b5k6QZxogALe for ; Wed, 18 Apr 2018 18:26:33 +0000 (UTC) Received: from mail-oi0-f48.google.com (mail-oi0-f48.google.com [209.85.218.48]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 625795F340 for ; Wed, 18 Apr 2018 18:26:33 +0000 (UTC) Received: by mail-oi0-f48.google.com with SMTP id e123-v6so2471184oih.13 for ; Wed, 18 Apr 2018 11:26:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=qJ8G3RzRE06FfvC6OYcLuuB8Vx+AfziCiQPDzVuC0ZM=; b=Aq8Qtoc2Yux20YfbEfeQlGdWhe9VngznaIEMyaFr2+v5Lw7+ypZ5871LlXOR93f6Xi eCy3ITchMDSIbEk2Ua4Xk9fhAocUnajC/I8LcBRoUT2V9hHvsTJdgPkoA3Xe2t+P3YP4 u9JnaDCDDf98GJ08j3blu1DQTNNfUpgylc/NqFCLvw5bfgU+O+jxHsW6iBzmCr/gez3l o+7d1YOJKzTPYJPrI9uNRiSWyKabTKAMsqxHnha6XSXet1yVAng1LNrgWDbtut9Af7Bo UjW3YKBVQnBVpMa8/enn/AXy3iHA2ANVc43HHcZJitI/SYpSRPiCF+dx3euqfOqUzpOJ LbCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=qJ8G3RzRE06FfvC6OYcLuuB8Vx+AfziCiQPDzVuC0ZM=; b=G0G2wduD718ei01Pqg0bFAfnfhUVSnq591Gth/4S2eZOg4HA+qzbXUV+7VFSaLxjIT rZQp9ha+eHTgFqBPCqaDmEE4+dgvXkj5d6qnux58NaoFJGN1YO3LYAzleWfiLrxG9ctF EF2HIAK2xxUATOK15l1uDPUzkFicoCTvXy9c7cvqc9UOkRgRYT2ZeYbHLcQWIXF7gGww FsKOMbSv/utj7SCQ+QOrdzLcjJ5g3kGUjYORDKIPqHJOyQvDnc7eh2APBC5oZCG5RNfO Op3b9fSmGcHG5HDEdKDZlygtECPrDLeL2JffKbPnx2nIVT0UfZBDvL4pnjGuXsDuZmnX r77w== X-Gm-Message-State: ALQs6tA+tp0lK6j32CD7S/TxSAfuaiYd6IMTpAoEc+ReGD8/9qjbPDv6 GWNv7qayln4nWPM6AHwyY5oWbKSuM0Qx7n+re7I9teQn X-Google-Smtp-Source: AIpwx4/2Hw2QGQDv78rGP1AtczaX6cob6czjENGsVA9aqSvX/BHHfJb/q7SxlZ7h8qmuXp4RfSEYdoojM5uWeiM/Rp8= X-Received: by 2002:aca:c503:: with SMTP id v3-v6mr1855800oif.153.1524075992254; Wed, 18 Apr 2018 11:26:32 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a9d:1f2e:0:0:0:0:0 with HTTP; Wed, 18 Apr 2018 11:26:31 -0700 (PDT) In-Reply-To: References: From: Ruiqin Yang Date: Wed, 18 Apr 2018 11:26:31 -0700 Message-ID: Subject: Re: Bit confused about start_date and schedule_interval related to daily/weekly DAG To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="000000000000211005056a239804" --000000000000211005056a239804 Content-Type: text/plain; charset="UTF-8" Hi Kyle, The execution_date of the DAG run will always be lagged one day for your daily DAG and one week for your weekly DAG. Under the hood, airflow will calculate the execution_date and next execution_date of the task, and only schedule the task when the current timestamp is bigger than the *next execution_date.* If you need date other than `ds` or `ds_nodash`, you can explore the other default variables from here . Cheers, Kevin Y On Wed, Apr 18, 2018 at 10:56 AM, Kyle Hamlin wrote: > I'm a bit confused with how the scheduler catches up in relation to > start_date and schedule_interval. I have one dag that runs hourly: > > dag = DAG( > dag_id='hourly_dag', > start_date=days_ago(1), > schedule_interval='@hourly', > default_args=ARGS) > > When I start this DAG fresh it will catch up 24 hours + however many hours > have passed in the current day all the way up to the most recent hour. This > makes sense to me. > > Now if I have a daily DAG: > > dag = DAG( > dag_id='daily_dag', > start_date=days_ago(1), > schedule_interval='0 5 * * *', > default_args=ARGS) > > Starting this DAG fresh will run yesterday's execution. This is fine since > I use the execution_date (ds_nodash) to have the task be lagged by one day. > What I can't seem to wrap my head around is how I would get this DAG to run > for the current day. I've tried passing is days_ago(0) but the tasks never > seem to start? > > In addition to all that, I have a weekly DAG that must also use the > execution_date, but it needs the current weeks execution_date. > > *How do I get a DAG that is not hourly to have an execution_date of the > current day or week?* > --000000000000211005056a239804--