airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yifei Hong <hon...@gmail.com>
Subject Re: How to control the frequency that Airflow refreshes dag?
Date Tue, 05 Dec 2017 16:22:07 GMT
Andy, thanks much for the prompt response, this is helpful.

So scheduler_heartbeats_sec controls how frequently the scheduler will run
and refresh the dags, right? Then in my case as we retrieve data and
dynamically generate subdags, if the current run is still undergoing and
external data source changes (i.e. from 3 run request to 5), I feel the dag
refresh will cause problem and mess up the running dag. Will Airflow
support this kind of usage? Dynamic dag generation is one of the Airflow
features that I like the most.

On Tue, Dec 5, 2017 at 11:02 AM, Andy Cooper <andy.cooper@astronomer.io>
wrote:

> Yifei,
>
> This is because the DAG definition file is not stored in the database. The
> scheduler must refresh the display of the DAG and it’s tasks from the DAG
> file itself. This means that it will be refreshed every time the scheduler
> is run. The schedule_interval refers to the schedule at which the DAG will
> run, not at which it will be refreshed.
>
> I would recommend not refreshing the task_ids and instead use static
> task_ids to handle an array/ list of tasks.
>
>
> > On Dec 5, 2017, at 10:58 AM, Yifei Hong <hongyf@gmail.com> wrote:
> >
> > Hi, Community,
> >
> > We use the main dag in Airflow to retrieve data from a database, and then
> > dynamically create a list of subdags for parallel runs with
> LocalExecutor.
> > For some task_id, we construct from now() to ensure the uniqueness.
> >
> > One problem we encounter is that Airflow keeps refreshing the dag so that
> > the task_id gets updated accordingly, even though we did not schedule the
> > dag and only run it via backfill.
> >
> > Could you please help advise how we can control the refresh frequency?
> Why
> > does Airflow not follow the scheduled frequency to refresh?
> >
> > Many thanks in advance,
> > Yifei
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message