airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiajie Zhong <zhongjiajie...@hotmail.com>
Subject Re: [DISCUSS] Change all example_dags's start_date to specific datetime
Date Mon, 13 May 2019 10:57:24 GMT
I know catchup_by=true by default, but I think when user run example dag, their just want to
test how Airflow run, and thier would stop it manually.

But if user use start_date=airflow.utils.dates.days_ago(2) as their production dag would cause
unexpected behavior.

IMO, if we use fixed date in example dag, dag would backfill, but user could stop it manually,
and their would know that "startdate control when dag start", and user will use fixed date
to their new dag(template from example dag)

Best wish.
-- Jiajie
________________________________
From: Deng Xiaodong <xd.deng.r@gmail.com>
Sent: Friday, May 10, 2019 15:50
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Change all example_dags's start_date to specific datetime

Please note that catchup_by_default is True by default. Imagine what will
happen if you set these start_date to fixed dates ;-)


XD

On Fri, May 10, 2019 at 3:34 PM Jiajie Zhong <zhongjiajie955@hotmail.com>
wrote:

> Hi Airflowers:
>     I start this thread proposal to change all example_dags's start_date
> to specific datetime, including airflow/example_dag and
> airflow/contrib/example_dags.
> I proposal that because in
> https://airflow.apache.org/faq.html#what-s-the-deal-with-start-date
>
> > We recommend against using dynamic values as start_date, especially
> datetime.now() as it can be quite confusing. The task is triggered once the
> period closes, and in theory an @hourly DAG would never get to an hour
> after now as now() moves along.
>
> we suggest to set start_date to specific datetime, but all example_dags in
> code base are use dynamic values like `airflow.utils.dates.days_ago(2)` or
> `datetime.utcnow()`. Most of Airflow starter will use example dag as they
> template to create their own dag, so I think should change them in correct
> values.
>
> But if we use a specific datetime in example dag, for example 2019-01-01,
> starter will use them as template and confuse "Why my dag start run in
> 2019-01-01?".
>
> So, should we change start_date?
>
> Best wish.
> -- Jiajie
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message