airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laura Lorenz <llor...@industrydive.com>
Subject Re: why is this alwasys running? Should be once an hour
Date Wed, 24 Aug 2016 15:04:54 GMT
I don't think this necessarily answers your question, but one thing I
noticed is that you are using a dynamic start_date, when you should be
using a fixed one. From the FAQs
<https://pythonhosted.org/airflow/faq.html#what-s-the-deal-with-start-date>:

We recommend against using dynamic values as start_date, especially
> datetime.now() as it can be quite confusing. The task is triggered once the
> period closes, and in theory an @hourly DAG would never get to an hour
> after now as now() moves along.


More to the point, what specifically do you mean by "always running" and
"fires every cycle"? For example is what you are seeing a new task instance
with a new execution date every run of the scheduler i.e. from the Browse >
Task Instances UI?

On Tue, Aug 23, 2016 at 5:27 PM, David Montgomery <davidmontgomery@gmail.com
> wrote:

> even @hourly is not working.  Fires every cycle. wow
>
> On Wed, Aug 24, 2016 at 5:09 AM, David Montgomery <
> davidmontgomery@gmail.com
> > wrote:
>
> > I updated the dag.  In thje UI I see 0 * * * * in the schedule field
> >
> >
> >
> > default_args = {
> >     'owner': 'airflow',
> >     'depends_on_past': False,
> >     "start_date": datetime.now(),
> >     'email': ['test@gmail.com'],
> >     'email_on_failure': True,
> >     'email_on_retry': True,
> >     'retries': 1,
> >     'retry_delay': timedelta(minutes=5)
> > }
> >
> >
> >
> > dag = DAG('first_test', schedule_interval="0 * * * *",
> > default_args=default_args)
> >
> > node_0 = PythonOperator(
> >     task_id='isnewdata',
> >     provide_context=False,
> >     python_callable=checkfornewdata,
> >     dag=dag)
> >
> >
> > node_0_1 = PythonOperator(
> >     task_id='fetchdata',
> >     provide_context=False,
> >     python_callable=fetchdata,
> >     dag=dag)
> >
> > node_0_1_2 = PythonOperator(
> >     task_id='uploadtoes',
> >     provide_context=False,
> >     python_callable= uploadtoes,
> >     dag=dag)
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message